{
"request": "Review this jira ticket",
"references": [
"https://mozilla-hub.atlassian.net/browse/MZCLD-3461"
]
}MZCLD-3461 tracks Wesley Dawson's exploratory build of a "Quick"-like internal platform (inspired by Shopify's internal dev platform blog post and Mozilla's existing protosaur.dev service). The ticket has progressed well beyond a PoC: it went from a $200/~1-day prototype to a live, fast-rolled internal service at quick.mozilla.cloud with ~50 sites and active GitHub Actions integration, all documented in-thread via 8 detailed comments. The ticket is still "In Progress" and reads more like a running engineering log/design doc than a bounded, closable task — there is no explicit acceptance criteria, and several open architectural decisions (tenant/repo placement, DB scaling, source-control policy, protosaur deprecation, Runlayer/MCP integration) remain unresolved as of the last comment.
[
{
"capability": "jira.issue.read",
"args": {
"key": "MZCLD-3461"
},
"result": {
"key": "MZCLD-3461",
"summary": "Prototype of quick-like internal platform",
"status": "In Progress",
"description_text": "Looking into options for a v2 after from I see Karlof mentioned . Reading through that blog post I observe that protosaur.dev is ~90% of the first part of that design (IAP+nginx+GCS+fuse). Some salient differences are permissions segmentation (in protosaur you can segment who can write to a website) and single domain only vs. wildcard domain (though protosaur used to support a wildcard setup). The pieces on top of the original architecture don\u2019t feel particularly complex to set up so I\u2019d like to explore how much of this can be built trivially in throwaway/PoC fashion. The main observation is that they\u2019ve got ~$200/mo shared VM running the entire developer platform of 50k websites, though they reference e.g. BQ and Cloud SQL databases. Notably: nothing about Quick says anything about source code hosting. It\u2019s basically agnostic / YOLO (as is protosaur.dev ). They don\u2019t mention how e.g. you\u2019d share the code for your site but I can imagine it\u2019s basically complementary to any and all of our github improvement efforts.",
"comment_count": 8,
"comments": [
{
"author": "Wesley Dawson",
"body": "The PoC took ~$200 of tokens but <1day of engineering time to get mostly functional: https://quick.sandbox.nonprod.dataops.mozgcp.net/ I\u2019ll share the IaC that built this via some mechanism soon, but assuming you are technical enough to install and authenticate to gcloud (arguably a higher bar than what we\u2019re targeting but the target of shopify\u2019s platform) you should be able to pretty trivially create a site using this stack. https://github.com/mozilla/mozcloud/compare/MZCLD-3461?expand=1 is some example slop to push a website via GHA that could be integrated with the forthcoming static site v1 implementation. I used this prompt to generate the site: https://jbuck-oracle.quick.sandbox.nonprod.dataops.mozgcp.net/ There are also various test sites including a 3d game to test some of the capabilities from the blog post. There were a couple of manual config steps for handling OAuth similar to protosaur but they are one time operation. The backend system is currently entirely serverless (Cloud Run + python) but uses a couple technologies like Cloud SQL for state. There\u2019s a high chance that the websocket implementation will need to be moved to something more server-like than Cloud Run, but I did point it at and my tests so far haven\u2019t run into any issues. Shopify used a VM but we might be able to swing GKE if necessary. I will socialize this within Cloud Eng a bit to get opinions on whether it makes sense to continue exploring a quick-like internal platform, but so far I\u2019m pleasantly surprised at how easy this was to set up and hope its feature set is robust enough to handle most of our internal use cases."
},
{
"author": "Wesley Dawson",
"body": "Based on some initial sharing within Cloud Eng :jbuck and I have decided it\u2019s worth properly evaluating this as a preview feature, the next steps of which are roughly: Create a proper lightweight mozcloud tenant for expediency this was bootstrapped using / v1 sandbox methodologies In that tenant, give this a better vanity domain like quick.mozilla.cloud We\u2019re currently planning to continue to copy branding from Shopify since that\u2019s expedient and basically unimportant. Some things that should be decided or discussed: Binary distribution: e.g. hooking into mozcloud builds. Currently the client-side app is a go binary and distributed via a bespoke mechanism accessible on the same site as the service. This might be fine but currently the build and deploy processes are local to my system. What permissions segmentation might look like if we needed it e.g. if I wanted to build a small app to get approvals for and wanted to know an approval was actually from the relevant user and not spoofed, some additional guarantees on service integrity will be necessary google_storage_managed_folder is very likely the best option here GHA OIDC is currently allowlist but could be made very permissive if desired sandbox-infra or webservices-infra ? This app is for mostly vibey/nonprod stuff but that doesn\u2019t necessarily mean it\u2019s a sandbox itself as a MozCloud platform feature. I\u2019m vaguely inclined to put this in webservices-infra but open to opinions on this mozilla-confidential data access I\u2019ve shared this site with DE and plan to make its SA a member of workgroup:mozilla-confidential/data-viewers so it can query the data warehouse, including crucially for Cloud Eng stuff like mozdata.mozcloud for tenant and workgroup config Runlayer integration The current LLM stack is very basic and doesn\u2019t have access to anything useful. I wonder if we get e.g. Slack integration or on-behalf-of type access out of integrating with runlayer Figuring out what to do if we outgrow Cloud Run I have low confidence this system will not run into scaling issues as users begin to use it. Shopify uses a $200/mo setup with a VM. If we need to switch to something more serverful than Cloud Run we should figure out how easy it is to hook into GKE or whether to go the VM route like Shopify did Once this is built as a proper tenant we can begin sharing it with other teams in a preview capacity."
},
{
"author": "Wesley Dawson",
"body": "Lightweight tenant bootstrapping went mostly smoothly and there\u2019s a PR up to provision https://quick.nonprod.webservices.mozgcp.net/ based on my local dev deployment. While examining existing mozilla.cloud DNS delegation I observed that there are some general patterns we should probably implement to make it easier to correctly set up and delegate vanity/brand domains to tenant infra, since in my opinion backstage.mozilla.cloud and stage.backstage.mozilla.cloud are incorrectly being managed across tenant / project / tfstate boundaries. We also have ~10 domains in webservices that could benefit from centralized support for vanity domains. I have a draft locally of the generalized DNS pattern but will focus on getting quick.mozilla.cloud out in a way consistent with the forthcoming work without committing to anything other than specifying the pattern at a high level."
},
{
"author": "Wesley Dawson",
"body": "https://quick.mozilla.cloud/ is live and I\u2019ve copied all the sites over to it from dev. Dev automatically redirects to it for now but I\u2019ll be shutting that project down in the next week or two. I think we\u2019re basically in a state where we can open up testing to a broader audience than Cloud Eng. There\u2019s a lot of cleanup and service hygiene work that needs to be done including getting the code built in CI but depending on how things go it might not make sense to prioritize that work. I\u2019ve filed a PR for DE to approve adding quick to the user-facing data platform as a member of workgroup:mozilla-confidential/data-viewers but next steps here are to respond to testing feedback and to examine adding more integrations e.g. slack, runlayer etc. Of particular note on the MozCloud side is that we can probably give it access to prometheus the way we do yardstick which similar to data-viewers access may be useful to some users."
},
{
"author": "Wesley Dawson",
"body": "https://github.com/mozilla/webservices-infra/pull/11738 https://feedback.quick.mozilla.cloud/ We\u2019ve added a bit of documentation and prefix conventions based on early internal discussions. We\u2019re planning to open this up to folks on Monday for testing."
},
{
"author": "Wesley Dawson",
"body": "We\u2019re continuing to refine the quick experience and have consolidated the github-backed static sites approach into quick, mainly because we learned today that static internal websites get a randomly generated domain that\u2019s not very user-friendly and attempts to address that basically wind up with a quick like solution. Austin and I have gotten internal-GHUSER type sites working e.g. https://internal-hamidatmozilla.quick.mozilla.cloud/ ."
},
{
"author": "Wesley Dawson",
"body": "Also worth mentioning I found some days back which is another quick clone with similar experience to ours."
},
{
"author": "Wesley Dawson",
"body": "At Karlof\u2019s request we ended up fast rolling the site and so far responses have been positive. There are ~50 sites already and even GHA integration has been enabled for a repo: . To accommodate the fast rolling I have been using for development beyond the original PR description. Of principal noteworthiness: I ended up needing to implement DB connection pooling when usage picked up because Cloud SQL connections were taking ~4s. There\u2019s plenty of prospective work that could be done but unless there are critical issues or feature requests I plan to prioritize working on hack week ideas while we gather usage feedback. Among the prospective work: Moving the quick code to a different repo and standardize/productionize build and deployment Karlof is interested in better discoverability for non-MozCloud users to see the code in case they want to contribute features or ideas But: webservices-infra will soon be an internal repository which may reduce the value here somewhat I\u2019m interested in seeing whether it makes sense to reuse or fold into in any way, or at least adopt the same build processes / have the same repo standards While we have spacelift and the point of using Cloud Run is for quick to be entirely serverless, it becomes a bit difficult to manage code deploys relative to straight terraform or k8s with argocd if we\u2019re containerizng the code in a non-webservices repo. I plan to discuss options on how to approach this with Austin later. Monitoring and observability Granting Quick access to monitor itself and maybe the same data that yardstick has access to. I\u2019m vaguely wary of scope creep here and wouldn\u2019t want people to try to replace yardstick with Quick. Discoverability Summarizing and categorization of dashboards feels like an LLM-shaped problem. If we wind up with 1000s of these sites we\u2019ll benefit from some way to deal with the sprawl. Lifecycle Management This is unsolved territory as the Shopify blog post doesn\u2019t get into. Fortunately this is an area Cloud Engineers are SMEs in so I have no doubt we can come up with reasonable lifecycle management semantics. Site locking / access restrictions I\u2019m hoping feedback is overwhelming in the direction that means we don\u2019t need to prioritize this, but Cloud Engineers are similarly SMEs in this area and we can come up with ways to reduce risk with the open permissions model quick is based on Runlayer / MCP integration We met with freshness earlier in the week to discuss this and I don\u2019t have enough access to Runlayer to try stuff out on my own. But it seems worth looking into Runlayer as a centralized mechanism for getting access to upstream systems like Jira. is an example of a service that requires hooking up to Jira in a specific Atlassian integration to be configured. Deprecate protosaur It\u2019s basically a subset of what quick does with slightly better permissions semantics. The only non-Cloud Eng user of https://protosaur.dev/ is the performance team and IIRC they run their jobs from Airflow so we could reasonably migrate this service without them needing to do anything SA and GHA access Access to write to the GCS bucket is opt in currently but could trivially be changed to allow any repo in any org we allow to use OIDC to write Similarly, we could expand the GCP principles allowed to write to include all service accounts or at least some workgroup SAs to minimize friction to use the site Stronger steering toward git and github for source control from wstuckey: \u201ckind of like how every github gist is a git repo\u201d Minimally we can add a git init on quick init if git is installed Source control for platform-managed sites Most notably https://feedback.quick.mozilla.cloud/ , but technically there\u2019s a few sites I built for exercising APIs that have the source code sitting on my local machine. None of these are complex enough that they couldn\u2019t be trivially restored directly from GCS"
}
]
}
}
]{
"model": "claude-sonnet-5",
"input_tokens": 6997,
"output_tokens": 2129,
"estimated_cost_usd": 0.052926
}[]