A possible definition of algorithm is to look at it as a concept composed on one side of a task, the aim that needs to be computed, and on the other of a data structure. Both these two notions are of fundamental importance and still need a good specification in the perspective of a proper theory of algorithms. The neural network framework is proving to be a good setting for challenging some corroborated notion about the relations between algorithms and programs, we will be using it as a stepping stone for proposing a new definition of the concept of algorithm. Based on one of the main theorems of the Kolmogorov complexity, our main argument is that there are more neural networks that are not implementing an algorithm, than the opposite. Or in other words, there are more programs that algorithms. The main consequences being foremost that the concept of algorithm is characterised by its nature of fundamental epistemological unit, separating it from a rather successful interpretation of mechanical manipulation of symbols. Secondly, that a theory of task gains a crucial role in the definition of algorithm. Moving on from unanimously accepted mathematical operations to only loosely formalizable computations (i.e., Cats and dogs’ classification), it scales up the theoretical difficulty of formally define these tasks, forcing to look for a paradigm able to account for both. Lastly, from this setting more than ever emerges the deep reliance of the computation on the data structures built on sets. The role of these representations is inextricable and contribute to the epistemological nature of algorithms.