Two efforts to convert a Python program to a Texera workflow

[chenlica]

Currently there are two efforts to convert a Python program to a Texera workflow:

1. Using an LLM to do the conversion. It can leverage latest AI techniques, but doesn't guarantee the equivalence between the two "programs". It's led by @zyratlo and @mengw15. See the discussion [WIP] Python Notebook → Texera Workflow migration tool using LLMs #5959 .

2. Using compiling techniques to do the conversion. It can guarantee the equivalence between the two "programs." It's led by @carloea2. See this discussion Proposal: Statement-level Python-to-Texera workflow compiler MVP #8160.

These two approaches have their pros and cons. We can present both to users and let users decide.

I want to start this discussion to let the community be aware of the two efforts, and provide feedback, if any.

[sshiv012]

Thanks for laying out the tradeoffs. Since the LLM-based notebook migration tool is already merged and relatively stable, could we first clarify the user need that requires a second Python-to-workflow path? The compiler’s equivalence guarantee could be valuable, but I’d like to understand which users or programs need that guarantee, and whether the current LLM approach falls short for them.
Before we commit to maintaining both approaches, could we define a small set of representative use cases and compare the results, including correctness, usability, and maintenance cost? If the compiler addresses a clear gap, that would make the case for supporting both much stronger.

[zyratlo]

I wrote out a response in this comment #8160 (reply in thread) but to summarize, the we can't guarantee the accuracy of the LLM response so the experience is slightly different. We provide a window with the source code and a mapping to help the user edit the workflow afterwards. From my testing of the tool, the LLM rarely provides a complete usable workflow especially for complex source code. The main pros of a compiler one given the LLM already exists is the equivalence guarantee and also not exposing a user's code to an LLM if they want to keep it private or it has sensitive information.

We did a brief comparison between the LLM tool and compiler tool using the same set of source codes, and it reflected what we've already said. But I support doing a more in depth comparison if it justifies the existence of both tools.

[parshimers]

I think this is an important topic. Learning how to solve conflicts in the direction of the project that naturally will arise, in an asynchronous and professional manner in a public forum, is hard. Since I am new to the project, and I don't want to force a viewpoint, I will describe some things that happened in AsterixDB that kind of have this flavor, and perhaps the community can use these examples to inform the eventual decision.

• Initially in AsterixDB, we used RTrees for spatial storage. These came after BTrees. As it turns out, there is a method of using BTrees for spatial indexing. You simply use a space-filling curve to translate the 2d point into some index on a 1d line, and build a BTree on that. There was also another way to do this that I can't remember. Someone implemented all three of these designs, and benchmarked them. It turned out that RTrees were the most performant all along. Therefore a separate implementation of 2 other indexes was completely thrown out, and the only thing that was merged back into master from it was a minor optimization to not do per-page latching on immutable index data during search. The only real product of the effort was a research publication.

• I was helping a new contributor to work on a new task. They had already contributed a relatively large change that nobody else wanted to touch due to how time-consuming and tedious it was to do. One issue that at that point, had been an issue for a very long time in AsterixDB, was the CSV parser. It was written by hand, and was very difficult to follow and understand. It often gave errors that didn't make a lot of sense. Since many of the initial uses of AsterixDB involved CSV this was a particularly acute issue.
  I was helping him to basically implement an idea I had, which was to simply use some other CSV parser from a library, rather than trying to fix the existing one. A similar thing had been done to great effect for parsing JSON, instead of using the ADM parser which was likewise handwritten. We got a few weeks into the task, when some other contributors wanted to discuss simply fixing the other CSV parser. They wanted to use it relatively immediately. They also had many other committers who were quite experienced and ready to work on it basically full-time, whereas the contributor I was helping was a student and hence only had some number of hours per week to dedicate to the task. Even though we had made some progress, worked openly, and had issues filed, I couldn't convince enough people to go with my approach and work together. The contributor basically had to backtrack, which was a very uncomfortable and disappointing thing. He did manage to make a great contribution after that, that is actively used to this day, so it ended up OK. It was a bitter pill, however.

[chenlica]

Some of the contributors had an offline discussion about this topic. I want to share my thoughts on this topic:

• The two approaches have their own pros and cons. The compiler-based approach has the benefits of guaranteeing the equivalence of the translation, keeping the user's code private, and having a high efficiency.

• It's not on the "critical path" of the engine, so we can be more tolerant of bugs.

• A significant effort has been put into the design and implementation of this compiler-based approach (about two people one summer).

• It does have certain restrictive assumptions.

• I understand the overhead of reviewing this feature and relaxing those assumptions.

• Having said that, I am OK with the plan (to be expressed by other contributors) that we want to first collect users' feedback on the LLM approach before deciding if this compiler approach is really needed.

@carloea2 @mengw15 @zyratlo @sshiv012 @parshimers Please share your thoughts based on our discussion. Thanks.

[mengw15]

I agree with this plan. We can first collect user feedback on the LLM-based approach, especially its limitations around correctness, privacy, and conversion quality. Based on that, we can better evaluate whether the compiler-based approach addresses a clear user need and justifies the additional maintenance effort.

[sshiv012]

Since the LLM-based approach has already been merged, I think our immediate priority should be gathering user feedback and improving that experience. We should let concrete user needs guide whether we maintain a second conversion path.

Privacy and security requirements are a separate consideration that we can evaluate when users raise them, including whether model providers offer appropriate compliance assurances and deployment options. I would avoid making those hypothetical requirements the deciding factor today.

I support continuing the compiler work in a fork, where it can mature and be evaluated. I appreciate the substantial effort already invested, and I’m happy to help review and evaluate it as it improves. Preserving that work gives us a valuable option while we learn more from users.

I’m open to bringing it into Texera once we have a demonstrated need, a mature MVP, and a clear justification for the ongoing maintenance cost. My emphasis would be on user experience, correctness, and code maintainability. A concrete next step would be to agree on a benchmark suite, an evaluation owner, and a review date, alongside collecting feedback on the LLM approach. That gives the compiler effort a clear path forward and lets us make the merge decision based on evidence.

[zyratlo]

I agree with the plan. Since the LLM tool is already merged, gathering feedback on it first makes sense, and that feedback should show us pretty directly whether users need the equivalence guarantee or the privacy the compiler gives.

I also agree with Chen's point about the critical path. Both the LLM and compiler tools are fairly separate from the engine, so a bug in either one doesn't have big consequences for the rest of Texera. That makes me comfortable keeping the compiler maintained outside main for now, as long as it stays current enough that merging it later isn't a big effort if the feedback shows a need.

I agree that since a significant amount of time and effort has been put into the compiler, we shouldn't abandon it. Putting it on a different fork and maintaining it there sounds like a good compromise. Since the current compiler implementation is still a MVP and doesn't support translating many Python capabilities yet, I wonder when it would be ready to start testing and benchmarking on real programs.