dimanche 9 juillet 2017

Feedback loop please

I’ve been working on data transfer processes these days. The transfer time is ridiculously slow regarding the amount of data (about 100k records). By the way it is geographic data, so it might be normal. It irritated me though and made me think about feedback loop.

The importance of shortening the feedback loop between coding and watching the result is huge.

A short feedback loop helps you to achieve more in less time. It also prevents you to get bored and fall into procrastination.

Days spent waiting for feedback are long and make me feel depressed. Days when I get quick feedbacks are pretty cool, whatever I am doing.

Some idea to shorten feedback loop :

  • TDD of course makes you write code that can be tested faster. Actually, taking much time to write test or wait for the test run to finish is a sign that you do something wrong. However TDD is not sufficient to test your whole app and you have to add system tests. They last longer.
  • Avoid network. Your network is unreliable. Use a local database on your machine when it’s possible. It also will prevent to crash the shared database for respect to your colleagues.
  • Automate what can be: to load your dataset, check the initial state of the system, test, cleanup.
  • Reduce the amount of data you will handle during your tests, at least for early stage of development
  • Ask for faster hardware in last resort.

This blog post is can of a truism. At least it will remain a note to myself!

vendredi 14 avril 2017


By skimming Python documentation today, I realized that logging module provides an exception level. Its goal is to show you a stacktrace when you call it from an except block.

You may have noticed that if you log an exception with level error, you only get exception message:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import logging

def send_unexpected_exception():
    raise Exception("I did not expect this")


except Exception as e:


ERROR:root:I did not expect this

Now use exception level:

It returns:

ERROR:root:I did not expect this
Traceback (most recent call last):
  File "logging_exception.py", line 12, in <module>
  File "logging_exception.py", line 7, in send_unexpected_exception
    raise Exception("I did not expect this")
Exception: I did not expect this

I think it’s an interesting way to provide feedback and context if your app raises an unexpected exception.

mercredi 18 janvier 2017

Accept my ideas

I like to question everything when I land into an existing team. As a normally pretentious person, I like looking smart, making people feel I'm experienced and that they can count on me. Yet, when I say what I find bad in a code base and when I expose my brilliant ideas, my new teammates don't follow me. That hurts my pride so hard!

“we've done this until now, there is no reason to change” says the lead dev. Of course they are reasons change, since the need for software and our knowledge evolve all the time! Questioning things is a good habit.

I've given up trying to look smart. It's being a jerk and lead to nothing. My solutions are meaningless when there is no problem, at least when the effects of the problem have not occurred (yet). So I state the problem, just to see if it's already known and if there's a consensus to do something about it or not. I do not insist. We decide not to do anything for now? That's OK, let's wait for occurrence. I'll offer my help when it will be needed.

The context is important also. First, we are contractors and we bill every change we make to the code. If we find a bug that the clients has not faced yet, it's complicated to inform them and demand money to fix it. It would be like maintaining a ransomware! It's a bad way to do software but the business is OK with that illusion of cost control.

Here is another factor. In our company, people are evaluated for their personal performance, not for the whole team performance. The benefits is that it theoretically push everyone to do its best and be rewarded for it. The problem is that it can easily create unproductive competition between people.

As a new guy in the team I have to figure out who are the leaders. It's useless to get in a fight with them. I propose stuff, and accept that they will be dismissed 80% of the time with a sarcasm. By the way, using sarcasm is lacking factual arguments. It's a good sign!

mercredi 4 janvier 2017

Pair programming

I appreciate pair programming more and more. It was a controversial practice for me. I used to find uncomfortable to share my computer, my tools, my personal universe with someone. Yet this practice is really powerful. Benefits:

  • It brings more focus. Having a pairing partner, I'm less prone to procrastinate. Even better, I feel like we are less likely to be disturbed by other people.
  • You catch more typos before running the compiler and / or the tests, hence save sensible amount of time
  • It's the only efficient way to share practices like TDD, shortcuts and habit that both pairing partners use to be productive
  • You practice code reviews on the fly, which is less boring that doing them later (you know later can mean never, don't you?)

It's not natural to feel comfortable with pair programming. Here are some advice I can think of :

  • coding dojo is a "safe" way to experience pair programming. However, they attract XP enthusiasts that may already know or use pair programming. You cannot force people to do coding dojo.
  • use a timer to switch and do breaks. Consider pomodoro technique. I use tomighty.
  • if you are more comfortable with pair programming than your pairing partner, pay attention to include her or him in the process. Force yourself to ask questions, ask how does he or she feel. Beware of silence. It's pair programming, not rubber ducking.

I know there is a more radical technique in the field today, called mob programming. I'm sure it has benefits in some thorny cases. As I am not a practitioner myself, I don't think it is something you do systematically. And I prefer learning to walk well before running.

What are you telling to your boss to do pair programming? Nothing. Just do it, and measure improvement. You may not be 2 times faster, but you will be faster and improve the overall quality of the code.

lundi 2 janvier 2017

Code reviews

I have to admit that I made few code reviews in my career. This is bad. Some that I made and some that other dev made on my code focused heavily mainly on the style: comments, ordering of methods, etc. This is worse. You see, lots of bugs cannot be found using unit tests and even performance tests. Examples:

  • hard-coded environnement specific values, like server names.
  • bad usage of external resources, like opening database connexion in a loop.
  • shared connections being closed by users, causing invalidation of open cursors (OK, you can catch thiw one with performance testing).
  • local directory usage on distributed systems. …

I faced all of them recently. They appeared in QA when we were lucky. Some of them have been found by the client in production. All of them could have been caught by a correct code review.

Believe me, I will intensify code reviews. Maybe you should too for 2017.

lundi 12 décembre 2016

Corporate hackathon

Some month ago I participated to a hackathon organized by my company with a colleague. The goal was to deploy a service on a Raspberry Pi 3 that could ingest 1 million messages and provide a synthesis of posted data in JSON format.

It was an occasion to get out of routine work and test new technologies. Here are the solutions we thought of :

  • Python + Redis
  • Node JS + Redis
  • Elixir

I discarded Python pretty soon, because it was not performant enough. Node looked promising and we found cool stuff to generate the synthesis using Redis. We also tried cluster server, which gave us tho possibility to run 4 nodes processes. Results where OK on our laptops.

We were waiting for official performance test suite from the organizers 5 using [gatling.io][gat], hence implemented in Scala). It's been buggy for a lot of time and could not rely on it to validate our solution. We were also waiting to be provided with a RPI from the organizers. The final approaching, I decided to buy one.

The tests on the RPI were surprising in the bad way, I did not expect this difference of speed. From 5000 queries per second, we were accepting about 3 times less.

We had another surprises when delivered with the final version of the official test suite : our synthesis was wrong!

I spent 2 nights before the final to deploy a version based on node and postgresql to fix this, as I was sure to be able to compute a synthesis in one SQL query. The request rate dropped bellow 1000 though, which meant we would be able to ingest 1000000 messages in more than 15 minutes. It's pretty long.

Other default, the JSON generated by node was not in expected format regarding numbers. Actually, I put an ugly hack at 3 AM to be able to serialize a Double with 2 digits of precision (you'll see it in our code, on node_pg branch).

The winner of the challenge based his implementation on Java using Undertow as a web server and MapDB for storage. The solution ingested the million of messages in less than 3 minutes and 10 million messages in less than 30 minutes when our solution failed miserably after 3 hours of processing! Actually, he has no problem for synthesis generation as the serialisation methods was exactly corresponding with test suite deserialization method.

So we did not as well as we expected, but we learned a lot:

  • Java is not a thing from the past, and concurrency based on multi threading was appropriate here . It's been a great reminder that the relevance of a solution strongly depend on the context.
  • We should have deployed on RPI sooner to adapt ourselves
  • Postgresql is a beast hard to tune if you don't have time to. I don't think its appropriate on a RPI. It seems hard to limit IO.

By the way, I'm happy with the results since we did not put so much effort in it and we still won a price (a Fitbit blaze that I'm currently trying to sell). Also, I did not have Internet at home at the time, so I did all my tuning offline! It was a great experience and we'll do better next time!

mercredi 23 novembre 2016

My talk at Pyconfr 2016

This year I got the chance to speak at the Pyconfr! It was motivating for me, as it is a national event and, as you may know, I really like Python.

I was a bit nervous, of course, but as trained myself before the talk, I gained confidence. It's a chance, because I was not so motivated to rehearse. But if I did not, my talk would have been lame. I mean, more that it actually was!

For the record, I submitted 2 talks at the CFP this year. The first one was about Hypothesis and property based testing. I really wished it would be accepted, as it would have motivated me to explore this library.

The one that has been accepted is Help, we to not have any Python project in my company, not so easy to present. You can watch the video here!

And, yes, it's in French.