Treat data science projects as software development, and you’ll avoid the most common pitfalls

Professional developers always strive to ensure the highest quality in data science projects. This may sound like a generalization, but it’s the truth. The data science craft is a non-trivial one – treated without caution, it might cause at the very least headaches, and in the worst case scenario, entirely misleading results. Best practices used by software engineers in their day to day work are applicable to data science as well and might spare you a number of serious problems.

Data science, as the name suggests, is all about exploring data and using it to harness information that isn’t immediately apparent. It can be a pretty exciting adventure! The tool which makes data science possible is programming – it allows you to dig through the enormous amount of data, create and train extremely complex models and most of all automate the processes.

This article is for you if:

• You want to find out about the most common pitfalls in Data Science projects
• You want to understand the consequences of handling these projects without caution
• You want to know what steps you need to take to minimise risk in this field

Why me:

• I work as a Python Developer in the 10Clouds’ Machine Learning Team specializing in data science solutions
• I have 5 years’ experience in analytics and data science – I know both sides of the coin
• My team has a wealth of experience in implementing data science solutions in production environments. We’ve worked on projects for companies across the globe, from start-ups to large-scale enterprises.

Key programming and data mining errors to avoid

Neither programming nor data mining are particularly easy. There are some common sins which are committed by people working in both disciplines, often leading to pretty serious consequences:

1. Programming alone, resulting in no external code review

Having only one specialist working on a particular project is never a good idea. There is no peer with whom to discuss problems, do brainstorming sessions and most of all to perform a code review. A code review is not just a way to double-check the code quality and correctness, but a method of optimizing a potential solution to discover alternative approaches – ultimately, its aim is to improve the code itself. A lack of peer review increases the probability of bugs creeping into the code. It may destroy the application or make the code fail silently and brake numbers. None of this is a desired effect. Code review should be treated as an opportunity of the personal growth of the developer.

2. A stronger focus on numbers than on the code itself

In data analysis, the data itself forms the core part of the project. That’s not surprising as data provides the answers to even the most difficult questions. However if programming is being used to conduct these analyses, focusing on code gives a significant advantage. At the beginning it may seem like a waste of time, but after a while the benefits will be visible – it will be easier to maintain and introduce to the new developers, many unintended flaws will be avoided. The power of programming offers the possibility to run countless experiments thanks to the automation of the processes – in the end, it makes the journey into numbers much wider, deeper and more fruitful.

3. Using tools designed for exploring data or prototyping moving toward regular development or production

There are a lot of useful tools which enable data scientists and developers to conduct experiments and test new ideas efficiently. One of the most popular ones is Jupyter Notebook – perfect not only for prototyping, but also for visualizing and presenting the code and results. It is good to use it for experimenting, but then to implement the chosen solution with tools enabling developers to share and maintain the code in future.

The purpose of it is to ensure that code is readable and easy to run for others – it’s worth to implement it by the book as it makes cooperation much more efficient and straightforward. Along with the code, it is worth to take care about the documentation and version control system. Documentation must at least explain how to run the code which includes a setup of the environment. It should also explain how the main features and functionalities work as these allow the project to live and keep its original ideas. The version control system gives a real power when it comes to sharing the code and keeping track of the changes made in application. These are very formidable ways of working with code, ensuring the comfort of cooperation and maintaining projects.

So how do we work at 10Clouds? First of all, we are treating data science projects in the exact same manner as the usual software development ones.

Code review is a crucial, inseparable part of development. No changes can be merged into the code without checking it previously with another developer. We definitely live according to the mantra that two heads are better than one. We ensure that peer code reviews are conducted at every stop, therefore reducing the risk of any potential bugs creeping in. It also gives us the opportunity to discuss and potentially review the solutions to the issues we’re working on and it is the best and easiest way to learn something new!

The data part is very important for us, but the main focus is still on the code. This means that all the rules and good practices which we use in developing software are applied to the data science solutions too. We pay attention to the cleanliness and readability of the code, thus it’s easy to maintain or to hand over to a new developer. We test the developed solutions manually as well as automatically in order to find and eliminate potential bugs or vulnerabilities. We know that if the code is written properly, then we can rely on the produced results.

For data science projects we use the exact same tools we’re using in our day-to-day software development work. These tools don’t just support the language we use – they also give us plenty of instruments with which to eliminate programming errors, make use of services such as Docker or Celery, and much more. Keeping track of code changes with version control system is therefore simple and supports an efficient code-review process. Documentation is a must have.

Are you looking for a team to work on a data science project?

We are great at what we do, and what we do is software development. Having data science knowledge and treating data science projects primarily as a software development process makes us your go-to team for data science products.

Would you like to read more about Machine Learning? Find the links below:
7 Things a Startup Should Know About Machine Learning
How Do You Fight Smog with Machine Learning? We Tried, and This Is What Happened

Get in touch with me on joanna.wilk@10clouds.com or visit www.10clouds.com. We have the expertise you need to make your product excel.