Tag
Data Quality on The Latent Daily: 1 story and 2 prompts. Latest: How to read a dataset's documentation: a datasheet checklist for AI researchers and builders.
Datasheets and data cards reveal what a dataset really contains. A step-by-step checklist for spotting gaps, bias, licensing risk and leakage in AI data.
thelatentdaily.com Editorial Team5 min read
You are a data analyst documenting a dataset so other people can use it correctly. Create a data dictionary for the table below. Table name: [TABLE OR FILE NAME] What one row represents (if known): [e.g. "one order line", "one user per day", OR "not sure"] Where the data comes from: [SOURCE SYSTEM, EXPORT, API] Who will read the dictionary: [e.g. "analysts new to the team", "business users building reports"] Schema (column names and types): """ [PASTE SCHEMA, DESCRIBE OUTPUT OR HEADER ROW] """ Sample rows (20-50, with sensitive values masked): """ [PASTE SAMPLE] """ For each column, give a table with: - Column name - Plain-English definition - Data type and format (units, date format, time zone if visible) - Allowed values or range observed in the sample - What a null or empty value seems to mean - Example value - Quality warning (mixed formats, suspicious defaults like 1970-01-01 or 0, duplicates) - Confidence: confirmed by the sample / inferred / guess Then: 1. State the likely grain of the table and a candidate primary key, and how to verify it. 2. List questions for the data owner, ordered by importance. Base definitions on the column names and sample only; mark anything inferred.
Works with: ChatGPT, Claude, Gemini, Any LLM
You are a careful data engineer. I need to clean a CSV file before analysis. What the data is: [SHORT DESCRIPTION, e.g. "customer sign-ups exported from our CRM"] What I will use it for: [THE ANALYSIS OR SYSTEM IT FEEDS] Approximate size: [ROWS AND COLUMNS] Tool I will use: [e.g. Python with pandas, R, Excel, SQL] Here are the header and a sample of rows (including some problem rows): ``` [PASTE HEADER + 20-50 ROWS] ``` Step 1 - Profile: for each column give the likely meaning, expected type, and problems you can see (missing values, mixed formats, inconsistent categories, outliers, duplicates, encoding issues, leading/trailing spaces). Step 2 - Plan: a numbered cleaning plan ordered by impact. For each step say what changes, why, and how many rows it might affect. Step 3 - Decisions for me: list choices that depend on business rules (e.g. drop or impute missing values) with your recommended default. Step 4 - Code: write [TOOL] code that applies the plan, keeps the original file unchanged, and logs how many rows each step changed or removed. Step 5 - Checks: list validation checks to run after cleaning. Base your profile only on the sample; say clearly when a problem may exist in rows you have not seen.
Works with: ChatGPT, Claude, Gemini, Any LLM