Most of the work in a data science project is cleaning data, not building models. A common rule of thumb says that cleaning takes most of the time, and experienced analysts treat it as the core skill rather than a chore.
Common problems to check
- Missing values: decide whether to remove, fill or flag them, and write down which you chose.
- Duplicate rows: check whether two identical rows are real repeats or a data error.
- Wrong types: numbers stored as text, or dates stored in two formats.
- Outliers: a value that looks wrong may be a real extreme, so check before you delete it.
- Inconsistent labels: “Bangalore”, “bangalore” and “Bengaluru” may be the same city.
Keep a log
Write down every cleaning step in order. A reader, or you next month, can then repeat the work or question a result.
Check the data again after cleaning
Compare row counts, totals and a few sample rows with the original file. Cleaning that silently changes the totals is a common cause of wrong results.
Next step: see our data science course, which includes hands-on data cleaning at ScholarLogic, or book a free demo class. A counsellor will share the fee and the next batch dates.
