Hiring a software developer who can take a lead role in organizing and structuring the codebase will also make the specialists better programmers, since they learn good programming techniques from someone who actually knows them.

Source: Senior software developer, where I initially started as a "algorithm/application developer" and saw the team grow and benefit by hiring some experienced developers.

All of that connecting tissue that makes things robust can take quite a bit of work, and it's really hard to see whether it's robust or not for a while.

I still think there are programmers more akin to artists who have a knack for solving complex problems elegantly. BUT there is a lot of craft in coding that can be learned. Even if you're not a greater programmer but you write clear, maintainable code you/someone else can improve it overt time.

There's nothing wrong with just writing code as you go and solving things while you find problems, but to get structured, documenting the behaviour of the program is a major point. Draw images how the code should work. What is connected to what, who owns what ?

By doing this process, you start to see the big picture. It's not always easy, and the tempation to just "go at it" is always there, but this easily leads to code you need to re-factor or re-think more than you would like.

But of course, sometimes you just have to write the crap version first, and then move on to a more advanced model. Actually, this is the expected way, as coding is a field where you just can't know what lies in that valley before venturing into it. So, accepting the fact that re-factoring is something you do constantly is also key. If you don't refactor, the old crappy solutions hinder and slow you down.

Experience of course helps you make better decisions in new projects.

Ill add a personal note. Being a good professional programmer is also about understanding the context of your program. You will not always have the time to optimize for performance. You will not always have the time to refactor functionality to make it more modular for further variety. Sometimes you will need to skip test becuase deadline says it should be done yesterday. Learn to not be emotionally tied to the code. Get your job done, to the best quality you can for the time you have allotted. Dont be made if its not perfect. But during the train ride or in the shower think about how to make it better. This process of just grasping with the ideas of clean code are good training for when you have to approach a similar problem on the fly.

As somebody has a built on top of the original BT codebase, it is not a well structured large program, and there is a reason nobody uses it for anything anymore.

* Code Complete https://www.amazon.com/Code-Complete-Practical-Handbook-Cons...

* Programming Pearls https://www.amazon.com/Programming-Pearls-2nd-Jon-Bentley/dp...

* Pragmatic Programmer https://www.amazon.com/Pragmatic-Programmer-Journeyman-Maste...

As for books, I would suggest looking at Head First Design Patterns by Freeman and Robson. This isn't explicitly about project design, but it is about designing reusable code modules which can really help in any language. It is aimed toward object-oriented languages, (the examples are in Java) but it has helped organize my code in other languages as well. If you're using the OO facilities of Matlab, VB and python this can be useful.

I came from a CS background and did a EE masters. Working in Matlab made me want to cry because of the problems you're describing; I think they're endemic to Matlab rather than anything you're doing wrong.

Source control will help, just look at some of the git tutorials. Go through them enough to where you can create a "release" branch that you can roll back to if things go wrong. You probably won't need or use the rest of the features right away if ever.

15k lines also really isn't that much. It sounds like you're managing it OK (separate files instead of one massive one, yay!) other than not having source control.

https://www.joelonsoftware.com/2000/04/06/things-you-should-...

Having your code base in python would be great, but do consider doing it piece by piece. Try to see if there's a part that would benefit greatly from the better organization features of python, and start there.

Matlab doesn't give you a choice here... every single function has to go in its own file.

I'm also not a huge fan of matlab.

I’ve seen this happen before in teams that code all day but don’t see themselves as developers. You need a instigate a cultural change at work where it becomes unacceptable to not apply modern development practices just because the team sees themselves as “quants” first.

I found Code Complete to be a much better book. It encourages a deliberate, thoughtful, sustainable approach to the structure of your code, and provides good examples.

Clean Code encourages you to decouple everything as much as possible, which in my experience leads to poor abstractions. (See https://www.sandimetz.com/blog/2016/1/20/the-wrong-abstracti...)

- Use version control!

- Hire a professional software developer

- Good architecture and abstraction comes mostly from practice but if you really want to do some reading on software architecture check out "The Architecture of Open Source Applications" (vol 1 & vol 2)

http://www.aosabook.org/en/intro1.html

Unfortunately it isn't in our budget to hire a pro dev to the team but I can work attempt to increase time working with our corporate IT department which has some excellent devs.

$ git init $ git add . $ git commit -am "Initial messy commit."

You could start using a more refined approach from there, and that would be a start.

When I start working with someone else's code for the first time, I run these commands and immediately have a baseline to work from. Once you run these commands, you can start experimenting with your code, and you can start guarding against someone else's accidental bugs in the project.

This does assume some basic understanding of how version control works, but it seems you could learn the basics of git in an afternoon and start using it to provide a baseline.

Are you currently doing the "make a copy of the project folder and call it project_backup_110217" approach to version control?

The fact that you make quant software trading millions of dollar without version control surprises me. Is the quant industry really like that?

Best way to learn personally is to get hired into a team that does serious project in the area you are interested in. If you are not looking to switch jobs, hire few people who have done that into your team, or get them on a contract (going to be crazy expensive but totally worth it).

First, basics, start with Joel's 12 rules of better code[1]

Next, buy everyone a copy of "Clean Code" book and read it from start to finish.

Look at the open source quant code bases, how they are organized and built. I suggest quantlib[4] and opengamma[3].

If you are looking at architecting quant library specifically, read this article by Russell Goyder[2]

[1] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s... [2] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2311743 [3] https://github.com/OpenGamma [4] https://github.com/lballabio/QuantLib

When you start using the same code in more than 3 places, put it in a shared lib type of file / folder.

When you start using the same code across 3 projects, put that into a "module" and manage independently in version control.

Depends on the language / framework you're using, but I've found that general rule pretty helpful in my experience.

For me I have a folder for the project, withing that a library folder shared code and libraries, one for the configuration files changeable templates, etc., and another that contains the applications folders (one is the main application one for login, user accounts, maint, etc.) then the different apps I make go into their own folders within the apps one. If you are using multiple languages you might want to have language sub folders as well.

Makes referencing the libraries and config info less crazy and segregates your apps for modularity.

Again check some similar project your language(s) might have differing file structure needs.

Example:

/project

/project/config

/project/lib

/project/app

/project/app/main

/project/app/thisapp

/project/app/anotherapp

etc.

EDIT: But for your immediate Matlab concerns I remember some StackOverflow questions about structuring large matlab applications. I just learned from one that Matlab has this notion of packages I was previously unaware of: https://stackoverflow.com/questions/2748302/what-is-the-clos... https://stackoverflow.com/questions/27861304/managing-bigger... https://stackoverflow.com/questions/2326609/how-to-visualize...

you may search StackOverflow for "matlab packaging" "matlab organization" "matlab structure" etc to get likewise hits.

Your first attempt is going to suck. How bad is a shade of gray, but it'll suck.

Accept this fact:

Your first attempt is going to suck. How bad is a shade of gray, but it'll suck.

And accept another fact:

Your third attempt is going to suck. How bad is a shade of gray, but it'll suck.

At some point, your attempts start to suck less.

Here are the major ones (that I'm aware of), making an effort to be inclusive:

1. Functional programming. It's possible I'm doing a "retcon" on several models here, but these are a fair start:

1.1. http://www.erlang.se/doc/programming_rules.shtml

1.2. https://www.seas.upenn.edu/~cis341/current/programming_style...

2. C (the language) organization. Embedded, linux kernel, and many open source projects follow this pattern. O'Reilly books are a good start.

3. Java. There are so many competing strategies that the "no true scotsman" fallacy applies. Here's one that seems uncontroversial: https://www.udemy.com/java-design-patterns-tutorial/

This point cannot be underemphasized.

There's no end of stuff you can read online about better practices, these days. But those experience I had working with people who, without so much taking "pity" on me, took me seriously, despite my obvious naivete in certain areas - I'm not sure whether I would have gotten anywhere at all without them.

The rest is going to be up to you and your team.

Don't try to enforce a structure unless it actually reduces the likelihood of bugs and improves the experience of refactoring.

The Pragmatic Programmer, by Dave Thomas and Andy Hunt, is a fairly slim book that aims to cover the unwritten knowledge that experienced programmers have. It came out a long time ago, but it's one of those timeless classics. If you can read this book and everything in it seems obvious, you are probably the equivalent of an intermediate-level software developer.

Similarly, Code Complete, by Steve McConnell, explains how to think like a project lead. In contrast to The Pragmatic Programmer, this is a large book: it aims to cover many concerns. Again, the less that you know, the more value that you will get from it: if you are starting with no training in programming, this book could be immensely valuable.

Re: high-level project layout: - where are their build files? - how are the source packages broken up? - where are the tests stored?

Trace back from the entry point to the code and look at how things are encapsulated, where code gets re-used.

Diagramming out with something like UML is cumbersome but can be helpful to visualize the structure as you're getting started.

Looking at software design patterns (while often overkill) can give you a good sense for some common ways people try to re-use code. For example: - https://en.wikipedia.org/wiki/Design_Patterns#Patterns_by_Ty...

- read other similar projects and understand how / why they work

- fix problems that you see.

If there are packages you're using from other systems that's often a very easy place to start with the first one... follow through all that code and see how it's organized.

With the second one, I'll say that it's really easy to put up with crappy workflows just because it is what you're used to. You have to not be willing to put up with the things you don't like in order to fix things. Use source control. Don't write functions that are 300 lines long. Think about the times that you have problems in your code and don't let those happen in the future.

A good first step is to get on top of your external dependencies. I find analysts tend not to think about this too much and happily just stick in whichever library, which can be a problem when you're working with a team.

So, make sure that your code base uses exactly one version of each library and everyone develops against that version. Next, make sure you only have one external library to server each purpose.

The R equivalent I've seen is a codebase which included dplyr, plyr (obsolete version of dplyr), data.tables (does the same job as dplyr, but with gibberish syntax), along with reshape (obsolete), reshape2 (obsolete) and tidyr. This sort of thing happens really easily because people want to use the tools they're familiar with.

Working with R has also led me to believe that some languages are just bad for writing large programs in.

So you may have to consider either a rewrite in a different language, or a rewrite so that you have several smaller programs connected with a glue language (e.g. `make`).

Either way, the software engineering part of software usually goes "extract a piece of this program, put it in a box, define the entrances and exits from the box carefully, then ensure your box does what it's supposed to".

The hard bit is deciding which way to cut the program up. That can have big consequences.

One way is to avoid duplication: e.g. if the same code appears in two different places, make it into a separate function, and call it from those places.

For general organization, unless you have enough experience tonknow how your code is likely to evolve in future, your organizational strategy may quickly become unsuitable. For the details of Quant Analysis, this may be particularly true. You may need a rougher, broader organization scheme (such as display as opposed to analysis), or a framework that provides the common support code needed by all analysis routines, so the analysis become less cluttered. Apply this even when the support code is only used by one analysis - so that its parts appear in different places.

But here there's a cognitive cost: that analysis may be harder to understand with its parts in different places.

The version control and testing seems simpler to start. You could start using git right away; but it is possible to muck things up - usually can be sorted oit, but there's a learning curve. But for such key code, it's probably another reason to hire an experienced dev, as others have said.

I read it in my late teens, and it's what made things finally click re: structuring programs. (This was back in the 90s, when it was far more difficult to inspect other well-structured software that wasn't also low-level C code.)

https://en.wikipedia.org/wiki/Structure_and_Interpretation_o...

Look at structural and behavioral patterns on https://en.m.wikipedia.org/wiki/Software_design_pattern

Facade, strategy , adapter, etc.

Also think about "levelization" John lakos explains that clearly.

Design from the top down; implement from the bottom up.

Write no subroutine longer than about 50-60 lines. Back in the day, that was a single printed page. We used listings because we didn’t have giant monitors.

Define a specific and single purpose for any given subroutine. Name it to match it’s purpose.

Don’t repeat yourself; got the same code in two places? Make it a subroutine.

If you don’t want to dive into git or other version control, use Time Machine or Dropbox to enable you to review what worked last week or last month.

Also, understand the worth of this code to your organization. 15k lines shouldn’t take a seasoned developer more than a few weeks to understand and refactor. If this code isn’t worth the $20k a good consultant would charge to fix it, then muddling through would appear to be part of your job description. There’s nothing wrong with slugging it out when economics dictate it.

Definitely have come along way from the first web app I wrote. Can't say it was not without a lot of writing echo 'test'; to see exactly where the error was, though of course, most frameworks tell you exactly where the error is now, but it helps with testing and debugging.

I do a lot more brainstorming in the beginning, designing of the web app, and then actual coding. It's really just practice. Learning how to keep your code organized and understandable for yourself. Write as if someone is looking over your code and isn't as smart as you :P

The structure of a program gets really bad when you continue to pile more and more code on a bad architecture.

Refactor early, refactor often - get to the point where you are comfortable with modifying large chunks. Of course that means you have enough test frameworks at every level.

Matlab as a language itself is more of a laboratory language, convenient for playing with math and quickly prototyping things.

As a language that can scale to dozens of thousands of lines, it is poorly suited.

You definitely need a senior programmer in your team, else it will soon turn into a nightmare.

https://www.google.com.au/search?q=how+to+structure+matlab+c...

Each time you encounter a new concept, search for that too.

Search for "beautiful matlab code"

Learn from others. Aggressively search for open source matlab code bases and read them.. look at their structure and try to understand why they are doing things the way they do... email the authors and ask them questions.

If you send me a message I would be happy to chat about it.

You can try to organize you stuff by: - layers (time) - functional units (what it does) - subsystems (parts assigned to people)

Over that, you can write a state machine to sequence the jobs, add tracing and isolate parameters.

The idea is to group, organize and classify iteratively.

Your project looks interesting.

Feel free to reach me @gmail.

Be wary of the glamour of "doing things the right way". Don't want to get into a new mess.

> Do you have a good book recommendation on project structuring?

Get it in source control and write some tests before you start hacking up the structure!

It's really not. Neither git, OOP, nor nice folder structures are a panacea (nothing is, of course). I've seen beautiful C in Source Safe and I've seen awful, overly engineered class happy crap in git repos.

Writing good, non-trivial systems requires experience and skills in many areas. No design pattern or source control system will save you from yourself if you're just not at that level yet.

I'm of the belief that most sloppy or "noodly" code is usually written that way not because the person is unable to produce good code, but because they just aren't trying. There are a lot of reasons why people don't try, and feeling intimidated by other devs is actually a pretty common one.

Maybe I'm just a slow learner, but I don't believe being shown a bunch of class Animal {}; class Dog : Animal {} examples really bring the point home. It's a combination of knowing what tools you have available to you and having the experience to know when and how to best use them.

In other words, gifting me a high end set of tools will not turn me into a great craftsman overnight. I think the "they're just not trying" argument is going to be hit or miss. Definitely true for some, not true for others. If you're 10+ years into your career and still writing unmaintainable spaghetti... then yeah, you're probably right.

1) Owning the project solely, in its entirety, from start to finish. this gives you that sense of responsibility that motivates you when you hit a road block or discover a major hurdle/roadblock/bug in your code

2) Creating large blocks of time to work on it. I found that when I tried to look at the code for 30-60 minutes at a time and get something done, that was the worst work I did and frequently had bugs or needed to be restructured entirely. The only way this didn't happen was when I was working frequently on the project. If I only had, for example, 5 hours in a given week to work on it, it was better to just not work on it unless those 5 hours could be sequential. It is hard to jump in to something, then think back to how it might affect something you wrote 2 weeks ago and then have to refer to that code. You'll spend more time just getting back up to speed than actually getting anything done.

For #1, this may or may not be possible for you. Not sure what the rules are about oversight and how well defined the roles are. But I found it extraordinarily helpful to involve absolutely no one. That also gave me a bird's eye view of everything and when you can recall how you structured a function/method/library that is being called, it's easy to see the pitfalls of how you will encounter a bug or hit your error capture section of your code. If you can't do #1 fully, then section off whatever you can. Clearly delineate what you are responsible for and what someone else is responsible for. For me, I hit limits with other systems that I needed to interact with within the company. So the maintainers of those systems were responsible for their system and only their system and I did my end separately.

For #2, this is usually very difficult to do. For the extremely large hurdles I needed absolute silence and huge blocks of time. This meant there were several weekends that I spent in the office, from 9 AM to 10 PM doing nothing but cranking out code with only food breaks. It worked for me, though. But it sucks big time for obvious reasons. Avoid that scenario at all costs if you can. Try to get those dedicated blocks at the office for complex coding tasks.

I would add a third critical element - that being to plan everything out in advance so that you can make sure what you have in mind actually works well. When you step through everything logically and see it all at once, you can identify problems, unforeseen situations, edge cases, and bottlenecks that will make you pull your hair out later on. Since this code already exists, I would instead modify this to be that you should plan out whatever solution you are going to pursue in full before you write even a line of code. Wait until you have a full game plan.

1) We can likely section off large components of code within the group and start refactoring/rewriting from there.

2) This can be tough on a trading floor but I do prefer to code in large time blocks as well.

Thanks again.

It's difficult to give you advice, since you've asked such a general question. I've written some thoughts on the matter, but take it all with a grain of salt. At best, my comments are spherical cows [0].

There's nothing wrong with having everything in a single folder; it really depends on the project. If you find it difficult to navigate your way around the source, I'd say there's two high-level approaches for organizing application code: you either group things by feature or by functionality. For example, many traditional web frameworks group source by functionality, so you have a folder for all models, controllers, views, configs, etc.

When structuring application components, you'll typically want to separate things into pure modules and anything which produces side-effects. Pure modules are typically easier to test and debug. If a module depends on anything, consider passing in any parameters during startup or when calling the module, rather than having it look things up on its own. That's the basic idea behind dependency injection.

Git is pretty much the standard for version control. You can get pretty far by just memorizing a few commands [1] and throwing up your code on Bitbucket, GitHub, or GitLab. To make life easier on yourself and your teammates, I'd suggest using a GUI tool like SourceTree. It makes it much harder to shoot yourself in the foot, while providing a far more approachable and discoverable interface. I'm not the biggest fan of git, but since it's what everyone else uses, you pretty much have to suck it up.

Since I'm not familiarized with the ecosystem I can't give much advice with regards to testing. The closest thing to a standard testing interface that I'm aware of is TAP [2]. I'd highly suggest checking out "An introduction to property based testing" [3]; it's one of the most insightful testing-related talks I've seen.

[0] https://en.wikipedia.org/wiki/Spherical_cow

[1] https://xkcd.com/1597/

[2] https://en.wikipedia.org/wiki/Test_Anything_Protocol

[3] https://fsharpforfunandprofit.com/pbt/

My recommendation would be to begin adding tests to your code, keep files sizes manageable (under 120 lines), keep functions under 7 lines. In general just be a professional instead of running around trying to fight fires all the time. Luckily 15k lines of code is a pretty small project so you aren't completely screwed.