Creating a +10,000 Line Python Package

Introduction

I started working on my own implementation of an electro-magnetic simulation package like MEEP¹ and, at the time of writing this, I am still working on it (so there is a lot still changing and a whole lot more to do). Check out BEAMZ on GitHub if you are curious. During the early development, I noticed this package becoming incredibly bloated very quickly so I want to lay out a few lessons and methods I used to make progress and write good code.

I’ll first walk you through the essential tools and workflows that helped me get started, from choosing the right IDE and learning Git to collaborating with AI. Then we’ll dive into some of the lessons of the actual process of scaling from simple scripts to a structured 10,000+ line codebase, covering some object-oriented programming, refactoring, file organization, and keeping track of everything as complexity grows. Finally, we’ll look at quality considerations like documentation, testing, and some opinionated takes on Python coding standards that worked for me in this project.

Getting Started

The Integrated Development Environment

When I began programming many years ago, I used Sublime² plus the terminal. Eventually, I switched to VSCode³ when I started working with others and to make it easier for myself to push and pull commits to and from GitHub and GitLab. And for about a year now, I have been using Cursor⁴, a fork from VSCode but with far superior AI integration.

Get Used to Git: Version Control

Most people in physics have never written code longer than maybe 1,000 lines and aren’t used to working with more than one or two files. Often, they simply work with jupyter notebooks⁵ and don’t really know anything about modules in Python let alone data structures, algorithms, or version control.

I have spoken about this before on Reddit but, in short, when you are co-programming with an AI and allowing it to write thousands of lines of code even, creating files, deleting your code even, what you need is version control. You need Git⁶. Make Git your ally. Git may not like you. But you need Git. So learn it!

What it allows you is to log your progress in time as you develop your project and revert back to previous iterations in case something went wrong. Also, you can create branches, new version of your code in a way, where you can go wild and experiment with it, then later merge it again with the main version of the code when you are happy with your additions. This allows for others to still build upong the existing, presumably working code, and for the team to add new features in parallel.

Like others said on my Reddit post as well, I recommend you install the Git Graph add on⁷ for VSCode/Cursor as it helps visualizing your git history.

Co-Working with Aritificial Intelligence

A great help to me early on was to go back to ChatGPT, send it some screenshots of my current branches and commits, explain what I want to do, and then let it explain to me. After a couple of repeasts of this, I eventually understood how to work across multiple branches.

The progress in AI is so rapid and quite noticable if you work with software to the point where many state that software development will soon be fully automated. Who knows? The only thing I know at the moment is that the way I collaborate with or use the AI is changing every month so there really is not much lasting wisdom to be shared here. I am just figuring it out, too.

I do want to say though: use it! This is expecially true if you are a domain expert e.g. a scientist but not a programmer! This will give you super-powers and finally bridge the gap that has so long existed between software engineering and the other sciences. It is amazing.

If you do know programming though, you should still use it! Use it for everything you do not want to do. Always try to have an agent running and working on something to maximize productivity and learn to manage your AI agents and review their code-submissions.

The Journey to a 10,000 Lines

When you do so, when you have a somewhat clear plan of what you want to build and you can communicate it well, then your AIs will quickly be able to build you a working version of this. But they may also do a lot of things worng or not precisely the way you would like for it to be done.

From Scripts to Modules

In Decemeber 2024, I had started playing with various scripts to create inverse designs for photonic devices⁸. A few months later, I started writing my own scripts to implement a custom FDTD simulation as well as my own adjoint solver. Both is fairly easy, can be done one a quiet evening and doesn’t require more than 10s-100s lines of code.

Object Oriented Programming

There is a lot to say here. I’ll keep it simple and try to say the most important thing first: Do not access the data of your classes directly. But let’s go through this a bit more systematically…

Object oriented programming is all about defining modules that have certain aspects, certain data as their properties, and providing methods, functions within these classes, to manipulate this data within the class to compute something or share it with other objects. This allows for clear modules and, ideally, safe modules that will not break your entire codebase if you change anything about them since everything is neately chunked into intuitive and largely closed / safe systems.

Accessing Data

Thing is… you can access the properties of an object. And in fact, you’ll often see this being done. The problem with that, if you directly access the properties of your object in other code, you are quickly falling into the trap of just duplicating it from one class into another and then maybe another and so on while also operating on that data. It all becomes mixed and tangled. Not only does the line-count quickly explode but you also lose track of where what is happening!

So lesson number one is to keep the properties of an object safe from the outside and write dedicated methods for when other aspects of your code are attempting to read them or manipulate them. They should be manipulating the object or prompt the object to do something.

I imagine it not like an object but like an alive animal. You don’t want to go inside and rearrange its organs. You’ll kill it! If you want your dog to sit, you tell it to sit. I.e. if you want your object to give you something or do something, there should be a method in that class to do that thing for you which you can call.

Inheritance & Abstraction

I believe the reason for why people tend to hate OOP is because of abused inheritance and premature abstraction. While it isn’t good to repeat yourself and write spaghetti code, it also isn’t good to go ahead and write a bunch of abstract objects way ahead of time of actually knowing what you are going to need precisely. This works for when you are creating a big new module but can quickly go wrong for writing smaller objects or sub-modules within the larger system.

A lot of legacy code-bases seem to have very complex networks of objects calling upon each other or, worse, deep hirarchies of inheritance. One might think that code can be radically simplified by creating general classes which are then reused to define more specific objects. Example: You might have an animal class with the general properties that animals have and then you’ll define a bunch more classes for specific animals like tigers, lions, elefants and so on which differentiate themselves with additional properties and methods.

The problem with this is that the deeper you go, the less is shown to the programmer within that class since it is defined elsewhere. One jump to a different class is okay. But two or three can become a nightmare.

Rage Refactoring

After a while, even with everything working, your code will likely be a Spaghetti of repeating elements, a bunch of dead functions that are never used, et cetera. In the case of BEAMZ, much of that code could probably be deleted or rewritten to be much simpler and easier to understand. Importantly, what I just explained about OOP I am breaking still. So I hope to correct these errors soon.

But I believe it is also just part of the natural development cycle that you’ll first manually add some code to just get something done, then the complexity increases and you realize later you could structure things more efficiently and elegantly by creating a dedicated class or method for it to be reused.

Refactoring is natural and should be done frequently. It can feel terribly uncreative and unproductive though which is why I personally find it dreadful to do. This, so far, has often been a bottleneck for my process. But it needs to be done!

File-Structure

Similarly, as the code grows and you keep adding more objects and methods, you’ll have to eventually split things into seperate files! It is generally good idea again to keep things accessible but brief. Meaning, your individual python files should maybe have 500 lines, maybe 1000 lines of code. But if you go beyond that, I doubt that this file is still only containing the tools for just one task. And this starts making reading the code difficult again since now you have to jump up and down in this single file without really knowing in which line something is. To avoid that issue, and keep conceptually different systems seperately, it is wise to add more files and split the functionality such that you can just go to the file whose only purpose is to do that one thing. For example, you might have a design file to define 3D structures and modify them and you might have a visualization file which provides all sorts of functions for the design file to use to visualize these structures!

Keeping Track with PyCodar

After I hit the 4,000 line count within the core package with many more thousand lines outside of the package for examples and various test and setup files, documentation and notes and so on, I was beginning to lose track a bit. There were now more than a dozen modules some with a thousand lines or more and many functions.

So I went on a little side-quest one evening and created another little package I named PyCodar, a very small one, which simply prints out useful information to me in the command line like an overview of the file structure but also listing all the classes, functions, and methods. Or giving me a line count or finding dead code. There are other tools out there like this but this was the first helpful solution for me and it is quite easy to use and prints out pretty results!

System Architecture Diagram

Aside from these stats, meta data and file structure overviews, what’s actually interesting of course is how the modules actually connect. This becomes increasingly important as the team expands or as you try to explain to the users of your software how it works and how to use it.

So far, my solution has been to simply use Excalidraw to map out the modules as boxes and their methods as arrows to show how they interact with each other. There are more sophisticated tools (e.g. Unified Modeling Language Diagrams⁹) for this sort of thing but you’ll probably be alright while still under 10,000 lines. I have been.

Quality and Packaging

Comments

When I started this project I thought I wanted to keep the line count as small as possible at first to keep track of everything. As a result, I made the conscious decision to not write any extensive comments, especially no multi-line comments under every little function to explain its arguments and returns and how it works, its history and whatnot… At this point, I am writing the code for myself still and I know what is going on. So to avoid bloat and keep things flexible, I actively avoided comments.

I believe adding comments should be done only once you start writing for other people. In the case of a project like mine, this only happened much later after many thousand lines were already written.

Testing

You might read a lot of posts online of engineers telling you to do test-driven development from the start, no matter what, especially now since LLMs can be quite effective when they have proper tests to check against. The idea of a human writing tests and the LLM just building the functionality to pass those tests is tempting.

I have tried this and my conclusion was that this makes absolutely no sense unless you already have some code established to properly guide you to the thing that needs to be built next. Maybe, if you have a lot of experience thinking in this way, you’ll find it natural and work more quickly. But I found it very unintuitive.

Clearly though, tests are incredibly important later on when you simply need to keep the entire code functioning!

Early on, I believe an example-driven approach is better. You write examples of how you want the user to use your code and then you start taking those functions apart and creating modules for them, step by step until your examples actually work. This is a lot like test-driven development but much more flexible, great for figuring things out when you don’t yet have all the ideas in place.

Documentation

To document the code, it is common to host webpages using GitHub Pages. I did the same thing. Another common technology is Sphinx¹⁰ where you are writing restructured text¹¹. I’ve had the great pleasure of doing this before and hated it. My love is markdown. It seems to me like restructured text is more flexible but also more complicated and Sphinx took many hours each time to render documentation. No thank you.

Instead, I went with and recommend MKDocs¹² which simply allows you to write markdown to create your documentation. I’ve been tinkering with getting Jupyter Notebooks to deploy on there, too. But that will have to wait for a future post, maybe.

On PyDantic & Types

I deeply hate the idea of strong types in Python. The beauty and power of Python lies in large part in its ability to just dynamically change the types of variables. If you really need to enforce your types, just define them in your arguments of your functions and write exceptions within those functions to check if the types and data-structure etc. are indeed as expected.

In many cases though, I think this can lead one astray and down the rabbit hole of premature optimization and create code bloat.

PyDantic is very useful for when you create an API. So at the highest level of the code which is directly interfaced with by the user, it may make sense to use PyDantic. And this is especially true for when you are communicating with databases and servers. But I see absolutely no reason to use PyDantic throughout the codebase.

Pythonic Code

Similarly to what I noted on comments, I believe that the code standards for Python are not necessarily always the best. It wastes a lot of space. When you have a lot of little functions or quite boring, easy to read functions with little complexity, it should not be necessary to have empty lines for seperation with comment headers over every operation.

Devil that I am, I also like to create single-line if-statements and similar quirks. Why write many lines when one line does the job? As long as it is easy to read (i.e. the line is still less than 90 symbols) and makes sense conceptually, I don’t see the issue with it per se. It does make sense to use a code-formatter though when, again, working with a larger team to ensure not every team member is introducing many of these weird little things into the code.

At least that is my perspective so far.

Conclusion

Most common wisdom for writing code does not actually apply when you are writing code for yourself rather than working on a large code-base within an organization. If no one else is reading it, it makes sense to move fast, to break and fix things quickly without getting distracted with too many standards only there to make collaboration easier.

As you appraoch a larger line-count and higher project complexity, you will need to start enforcing some standards and really think about your architecture though. Having a proper grasp of object oriented programming is essential.