How to evaluate AI app builders beyond the first demo

Do not rank AI app builders from a showcase prompt. Freeze one realistic fixture, start each tool under comparable conditions, and run a sequence: first build, two focused revisions, one failure-state request, verification review, and an ownership exit drill. Capture prompts, time boundaries, source, costs, manual interventions, and reproducible results.

Score the evidence on product correctness, revision containment, failure honesty, verification, source quality, portability, and cost predictability. A strong first render can still hide mock behavior or fragile ownership. A plain first pass may become the better system after precise revisions. This guide is published by Mythos, so it deliberately provides a vendor-neutral protocol rather than a Mythos ranking.

The first generation favors visual breadth. It rewards a tool that recognizes a familiar pattern, chooses attractive defaults, and fills the screen with plausible data. That is useful, but it is not the work that usually determines ownership cost.

Real product development asks different questions:

• Can the system preserve unrelated behavior during a focused change?
• Does apparent data persist after refresh and across identities?
• Are permissions enforced by the server or only hidden in the UI?
• Does failure stop honestly, or does the product silently simulate success?
• Can a human run, inspect, branch, and continue the source elsewhere?
• Are charges, retries, and external side effects understandable?

A benchmark that ends at the screenshot measures taste and pattern recall. A benchmark that continues through revisions, negative states, and handoff measures whether the tool can participate in a durable engineering loop.

Do not compensate by choosing an enormous “build a whole startup” prompt. Large prompts introduce more uncontrolled choices and make failures hard to attribute. Use one fixture with enough state, roles, and persistence to expose boundaries.

Write the job and constraints first. An individual testing landing-page ideas may value time to editable layout and easy export. A team building an authenticated operational tool may prioritize server authorization, revision safety, Git workflow, and auditability. One universal winner is not credible.

Create a weighted decision model:

The weights are an example, not a recommendation for every team. Freeze yours before seeing results. Otherwise a favorite tool can redefine “important” after each test.

The fixture below is small enough to repeat and rich enough to expose common shortcuts:

Build a support-request app for a small software company.

Users:
- Customer: creates a request and sees only their own requests.
- Agent: sees all requests, changes status, and replies.

Core path:
1. A customer signs in and creates a request with subject, category, and details.
2. The request persists and appears in the customer’s history.
3. An agent opens it, replies, and changes New to In progress or Resolved.
4. The customer sees the reply and updated status.

Required states:
- empty, loading, validation error, server error, success;
- unauthorized access to another customer’s request;
- duplicate submit protection;
- desktop and mobile layouts.

Constraints:
- use the tool’s normal recommended stack;
- do not claim mock or browser-only state is durable multi-user storage;
- do not add paid providers unless explicitly required;
- keep private configuration out of client source.

Deliver the running candidate and source needed for an independent handoff.

Use the exact wording in every tool unless a product requires a syntactic wrapper. Record any clarification the tool requests and answer consistently. If one tool needs an external account or backend, document that requirement rather than secretly giving it an advantage or penalty.

Create a run sheet before testing:

• tool, plan, model, and region if exposed;
• date and start/end times;
• new project or template state;
• prompt text and attachment hashes;
• optional services connected;
• credits or charges shown;
• manual edits, retries, approvals, and support interventions;
• source revision and Preview URL or artifact;
• observed result for each acceptance case.

Use fresh projects and the same identity structure. Do not import a polished starter into one tool and begin another from blank. If a tool automatically supplies infrastructure, record it as part of the product experience and inspect the ownership boundary.

Stop the clock when the same milestone is reached, not when the interface declares success. Suggested milestones:

1. first interactive render;
2. first complete critical path;
3. both revisions accepted;
4. failure behavior verified;
5. clean checkout builds independently.

Time is descriptive, not sufficient. Five fast minutes that produce simulated persistence should not outrank fifteen minutes that produce the requested durable path.

After the initial result, do not immediately repair it. Capture what the tool chose without extra coaching. Walk the fixture:

Inspect network and source where available. A card that appears after clicking Submit may be a local state update rather than a durable record. A role dropdown may be a visual toggle rather than authenticated authority. Mark an item passed only when the evidence matches the claim.

Also note the amount of irrelevant output. Extra dashboards, charts, and settings can make the demo look complete while increasing the surface that later revisions may break.

The second request tests containment:

Change only the request list and filtering experience.

Add search by subject and filters for category and status. Keep authentication,
request creation, agent replies, persistence, and all existing routes unchanged.
The filter state must survive refresh through the URL. Clear resets the URL and
visible results. Include empty and no-match states.

Verify the new behavior and rerun the original critical path. Record unrelated files or product surfaces changed.

The third request tests a semantic change across the system:

Rename the customer-facing concept “request” to “conversation.”

Change visible customer copy, accessible labels, empty states, and navigation.
Keep internal data identifiers, URLs, database shape, agent permissions, and
behavior unchanged. Do not run a migration only to rename customer-facing copy.

This reveals whether the tool can distinguish presentation from storage and whether it follows explicit invariants. Review diffs or source trees, not just the final Preview. Penalize silent architecture changes, broad rewrites, lost features, and hidden workarounds.

Do not demand the fewest changed lines. The correct root-cause change may cross shared components. Evaluate whether the change is coherent, understandable, and proportionate.

Create one safe failure you can repeat. Use the tool’s supported test controls when they exist. If they do not, use a non-production service or test fixture that can reject the request without causing a real side effect.

Use this protocol:

Make failed request creation explicit and recoverable.

When the server rejects creation, keep valid form values, show one clear error,
prevent a duplicate record, and allow a deliberate retry. Do not report success,
write a local-only record, or navigate away. Preserve the current successful path.

Observe whether the tool:

• identifies the operation that writes the data;
• preserves the user’s work;
• distinguishes validation from server failure;
• avoids automatic repeated side effects;
• keeps visible success and failure outcomes consistent;
• admits when the connected backend cannot be controlled in the requested way.

Failure honesty is a core product feature. A system that says “done” while replacing the requested behavior with a mock is harder to operate than one that stops with a precise limitation.

Ask three practical questions: what was checked, which version was checked, and what happens when a check fails. Useful evidence can include input validation, type checking, a production build, targeted tests, browser rendering, accessibility checks, and a smoke test in the intended environment. A longer checklist is not automatically better; every check should cover a risk you actually care about.

For Mythos, every delivered Build passes TypeScript and a Vite production build. Mythos also runs desktop and mobile browser checks when that verification is available; if it could not run, score those viewports as unverified. That is valuable implementation evidence, but it is not a claim of complete security, accessibility, or release readiness. Confirm the current behavior in Building and editing when you run the comparison.

Apply the same standard to every tool:

Do not count a generated message such as “all tests pass” unless you can see or reproduce the checks behind it.

A repository icon is not a handoff. Export or connect the project, then continue without relying on the original builder session.

The evaluator should:

1. obtain the full source through the documented path;
2. clone or unpack it into a clean directory;
3. identify the runtime and package scripts;
4. install dependencies and create an intentional lockfile if one is absent;
5. run the typecheck and production build;
6. provide fresh non-production configuration using documented variable names;
7. make a small change on an independent Git branch;
8. verify the app runs with standard tools;
9. disconnect the builder, where supported, without deleting the repository you own.

Record what does not travel with the source: hosted databases, authentication services, object storage, domains, secrets, deployment settings, and provider accounts. A good tool makes those boundaries clear instead of implying that a Git export copies third-party infrastructure.

When evaluating Mythos, budget for the plan needed by the test: GitHub connection and synchronization, project ZIP downloads, and manual code editing require an active paid workspace subscription. Code ownership itself does not depend on that subscription. Check pricing and the GitHub integration documentation, then test repository control, branch behavior, conflicts, and disconnect. Documentation defines the expected flow; the exit drill proves it works for your project.

Use one rubric for every criterion:

Multiply each score by the frozen weight, then attach evidence links. Keep raw observations separate from interpretations. “Build completed in 8 minutes” is an observation. “Best for rapid prototyping” is an interpretation that must account for correctness and intervention.

Do not publish a ranking from one run. Provider incidents, models, product versions, plans, and reviewer skill can change results. Repeat critical tests, disclose dates and configurations, and say when a claim is an inference.

A decision report should end with fit by use case, material limitations, and switching cost—not a universal winner. If two tools score closely, choose based on the risks your team can operate, not a decimal-place total.

One fixture cannot represent every application. It underweights domains such as realtime collaboration, native mobile, large migrations, media processing, payments, regulated data, and unusual deployment targets. Add a second fixture only when it tests a requirement central to your decision.

The evaluator affects the result. Prompt skill, willingness to inspect source, and familiarity with the stack can advantage a tool. Preserve prompts and interventions so another reviewer can challenge the result.

This guide does not contain current competitor rankings because no live, controlled comparison accompanies it. Product capabilities and prices change. Test the actual plan and version you intend to buy, and verify current claims in first-party documentation.

How many tools should I test?

Test the smallest shortlist that meets your non-negotiable platform, budget, and ownership requirements. Deep evidence from three candidates is usually more useful than shallow screenshots from ten.

Should every tool receive exactly the same prompt?

Keep the fixture and acceptance criteria identical. Allow only required syntax or setup differences, and record them. Do not quietly improve one prompt after seeing its failure.

Is time to first render a useful metric?

Yes, as one timestamp. Also measure time to a complete critical path, two safe revisions, failure handling, and independent build.

How do I compare different pricing models?

Record the plan, included usage, visible charges, retries, and external provider costs for the same milestones. Avoid projecting large savings from one small run.

Should visual quality have more weight?

Increase its weight if visual exploration is the buying job. Keep product correctness and ownership visible so a strong composition does not hide a non-durable system.

Can this protocol prove a tool is secure?

No. It can reveal inspectability and obvious boundary failures. Security assurance requires a threat model and testing appropriate to the application and operating environment.

Mythos behavior disclosed in this vendor-neutral protocol comes from:

• Building and editing
• GitHub integration
• Writing good prompts
• Version history
• Workspace editor
• Credits and usage
• Troubleshooting

The fixture, scorecard, and weighting method are an editorial test protocol. They are not results from a current competitor benchmark.

Continue with:

• From prompt to a production candidate
• GitHub handoff for generated apps
• Plan vs Build in Mythos