Add importer-provider extension capability for infrastructure generation#7452
Add importer-provider extension capability for infrastructure generation#7452
Conversation
…e importer list
Introduce the Importer interface that captures the contract for project importers
(CanImport, Services, ProjectInfrastructure, GenerateAllInfrastructure). This enables
extensions to provide custom importers via the extension framework.
Key changes:
- Define Importer interface in pkg/project/importer.go
- Make DotNetImporter implement Importer with Name() returning "Aspire"
- Refactor ImportManager from hardcoded dotNetImporter field to []Importer list
- ImportManager iterates all registered importers (first match wins)
- DotNetImporter.CanImport now accepts *ServiceConfig and checks language
before expensive dotnet CLI detection
- Move Aspire-specific constraints (single service, ContainerApp target)
into ImportManager's importer iteration with importer name in messages
- Update IoC registration to build importer list with DotNetImporter
- Update all test files to use new NewImportManager([]Importer{...}) signature
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implement the complete gRPC infrastructure for importer extensions: Proto & Code Generation: - New importer.proto with ImporterService (bidirectional Stream RPC) - Messages: CanImport, Services, ProjectInfrastructure, GenerateAllInfrastructure - GeneratedFile type for serializing fs.FS over gRPC Extension SDK (pkg/azdext/): - ImporterProvider interface for extension-side implementation - ImporterManager for client-side gRPC stream management - ImporterEnvelope for message envelope operations - ExtensionHost.WithImporter() registration method - Full lifecycle integration in ExtensionHost.Run() gRPC Server (internal/grpcserver/): - ImporterGrpcService implementing server-side stream handling - Capability verification and broker-based message dispatch - IoC registration of ExternalImporter on provider registration Core Integration: - ExternalImporter adapter implementing project.Importer over gRPC - Handles ServiceConfig/ProjectConfig proto conversion - Temp directory management for ProjectInfrastructure - memfs reconstruction for GenerateAllInfrastructure Extension Framework: - ImporterProviderCapability and ImporterProviderType constants - Added to listenCapabilities for extension startup - Updated extension.schema.json with new capability and provider type Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
IoC Registration:
- ImportManager now accepts ServiceLocator for future extensibility
- Added lazy ImportManager registration for gRPC service access
- ImporterGrpcService uses AddImporter() to register external importers
at runtime instead of IoC named registration
- Updated all test files with new NewImportManager(importers, locator) signature
Demo Extension:
- DemoImporterProvider detects projects via demo.manifest.json marker file
- Generates minimal Bicep infrastructure (resource group)
- Registered in listen.go with host.WithImporter("demo-importer", factory)
- Added importer-provider capability and provider entry to extension.yaml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test Infrastructure: - New extension-importer sample project with azure.yaml and demo.manifest.json - Integration test (Test_CLI_Extension_Importer) that builds/installs demo extension, copies sample project, and verifies extension starts with importer capability Registry: - Updated registry.json with importer-provider capability and demo-importer provider entry for microsoft.azd.demo extension Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ExternalImporter was sending RelativePath (e.g. './src') to the extension, but the extension process has a different working directory and couldn't find project files. Now sends the fully resolved absolute path via svc.Path() so extensions can access files regardless of CWD. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The ImporterGrpcService (singleton) was adding importers to a scoped ImportManager instance via lazy resolution, but the azd command action would get a different scoped ImportManager that didn't have the extension importers. Fix: introduce ImporterRegistry as a singleton shared between the gRPC service (which adds importers on extension registration) and all ImportManager instances (which query the registry via allImporters()). This ensures extension-registered importers are visible to any command that uses the ImportManager. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Logs the number of available importers, their names/types, and the result of each CanImport() call. Run with --debug to see this output, which helps diagnose why extension importers may not be invoked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The infra generate command was missing the extensions middleware, so extension-provided importers were never started. Added UseMiddleware for both hooks and extensions, matching the pattern used by infra create and infra delete. Also added debug logging to GenerateAllInfrastructure to trace importer availability and CanImport results. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace demo.manifest.json approach with a markdown-based resource definition format using azd-infra-gen/v1 front-matter header. Sample project (extension-importer): - infra-gen/resources.md defines a resource group and storage account with tags using a readable markdown format with YAML-like properties - azure.yaml points to ./infra-gen as the service project path Demo importer (DemoImporterProvider): - CanImport: scans directory for .md files with azd-infra-gen/v1 header - Parses resource definitions from markdown (H1 = resource, - key: value) - Generates main.bicep (resource group + module reference) - Generates resources.bicep (storage account with sku, kind, tags) - Proper Bicep string interpolation for env var references - Both ProjectInfrastructure and GenerateAllInfrastructure produce the full file set Note: azd init integration for extension-based project detection is not yet implemented. Currently extensions participate only at provision and infra gen time via the ImportManager. Init-time detection would require extending the appdetect framework to query importers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Architectural redesign: importers are no longer defined as services.
Instead, azure.yaml has a new infra.importer field with name and path:
infra:
importer:
name: demo-importer # matches extension-registered importer
path: ./infra-gen # path to importer project files
This keeps the services list clean for only deployable services that
can be built and packaged. The importer is a separate concern that
generates infrastructure, not a deployable unit.
Key changes:
- Added ImporterConfig struct to provisioning.Options (name + path)
- ImportManager.ProjectInfrastructure checks infra.importer before
auto-detection (backward compat with Aspire preserved)
- ImportManager.GenerateAllInfrastructure likewise checks infra.importer
- Importer interface: ProjectInfrastructure and GenerateAllInfrastructure
now take importerPath string instead of *ServiceConfig
- DotNetImporter wraps the new interface with path->ServiceConfig adapters
- ExternalImporter sends path via ServiceConfig.RelativePath over gRPC
- Updated sample to use infra.importer with infra-gen/resources.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The importer config now uses an open-ended options map instead of a
fixed path field, giving extensions full control over their settings:
infra:
importer:
name: demo-importer
options: # extension-owned, schema defined by extension
path: custom-dir # optional override, extension defines default
Key changes:
- ImporterConfig.Path replaced with Options map[string]any
- Added GetOption(key, default) helper for string option lookup
- Importer interface methods now receive (projectPath, ImporterConfig)
so extensions can resolve paths and settings themselves
- Proto updated: infrastructure requests send projectPath + options map
- ImporterProvider interface uses (projectPath, options map[string]string)
Demo extension updates:
- Default path is now 'demo-importer' (convention-based, no config needed)
- Reads 'path' from options to allow override
- Sample project renamed infra-gen/ to demo-importer/
- azure.yaml simplified: just name, no path needed
This illustrates the pattern: extensions own their defaults and options.
Users only need to specify the importer name. Options are optional overrides.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Demonstrates combining an extension importer with a deployable service:
Sample project structure:
azure.yaml - defines infra.importer + 'app' service
demo-importer/ - resource definitions (.md files)
src/app/ - static web app (vanilla JS/HTML)
dist/ - pre-built static files for deployment
package.json - no-op build script
azure.yaml:
infra:
importer:
name: demo-importer # generates all infrastructure
services:
app:
host: staticwebapp # deployable service
language: js
project: ./src/app
dist: dist
Generated infrastructure (resources.bicep) includes:
- Storage account with tags
- Static Web App with 'azd-service-name: app' tag linking it to the
service in azure.yaml, enabling azd to deploy to the correct resource
This shows the clean separation: the importer owns infrastructure
generation, services own build/deploy. They connect via azd-service-name tags.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ProjectInfrastructure was using SendAndWait which doesn't handle progress messages. When the extension sent a progress update before the response, SendAndWait received the progress message and returned nil for GetProjectInfrastructureResponse(), causing the 'missing response' error. Fix: use SendAndWaitWithProgress which correctly filters progress messages and waits for the actual response. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The demo importer now generates main.parameters.json with the standard azd parameter mappings (environmentName, location, principalId) for both ProjectInfrastructure (runtime) and GenerateAllInfrastructure (azd infra gen). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep only the resource group and static web app in the demo sample to focus on the essential pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New doc: cli/azd/docs/extensions/extension-custom-importers.md Covers the importer-provider capability, how it works, the demo importer as an analogy for real-world use cases like #7425, writing your own importer, combining with services, infra override/ejection, and current limitations (azd init integration). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jongio
left a comment
There was a problem hiding this comment.
Issues to address:
- dotnet_importer.go:625 - nil ProjectConfig passed to generateAllInfrastructureFromConfig causes panic
- importer_external.go:158 - extension-provided file paths not validated for path traversal
- importer_service.go:89 - stale importer left in registry after extension disconnect
- importer.go:411 - HasAppHost now triggers for non-Aspire importers, causing incorrect CI behavior
- importer_external.go:245 - noop env var resolver mangles resource names
- extension_importer_test.go:102 - test comments say "infra synth" but actually runs "azd show"
| if err != nil { | ||
| absPath = projectPath | ||
| } | ||
| svcConfig := &ServiceConfig{ |
There was a problem hiding this comment.
[CRITICAL] Nil pointer panic in GenerateAllInfrastructure bridge
generateAllInfrastructureFromConfig dereferences p.Infra.Module, p.Infra.Path, and p.Path - all panic when p is nil. This path is hit whenever ImportManager.GenerateAllInfrastructure auto-detects an Aspire project (the interface dispatches here through the bridge).
Simple nil guards aren't sufficient either - the inner method uses filepath.Rel(p.Path, ...) to compute service manifest locations. Consider constructing a minimal ProjectConfig{Path: projectPath, Infra: DefaultProvisioningOptions} instead of passing nil.
| } | ||
|
|
||
| for _, file := range infraResp.Files { | ||
| target := filepath.Join(tmpDir, file.Path) |
There was a problem hiding this comment.
[CRITICAL] Path traversal - extension-controlled file paths written without containment check
file.Path comes from the extension over gRPC and is joined directly into tmpDir without validation. A path like ../../.ssh/authorized_keys would escape the temp directory. Extensions are user-installed, but defense-in-depth still applies.
Validate that filepath.Clean(target) stays under tmpDir before writing.
| } | ||
|
|
||
| s.providerMapMu.Lock() | ||
| delete(s.providerMap, registeredImporterName) |
There was a problem hiding this comment.
[HIGH] Stale importer left in registry after stream close
Stream teardown removes the importer from providerMap but not from ImporterRegistry. The stale ExternalImporter stays in the registry with a dead broker - any subsequent call to it fails with a gRPC stream error instead of a clean "importer not available" message.
ImporterRegistry needs a Remove(name) method called here alongside the providerMap delete.
|
|
||
| // HasAppHost returns true when there is one AppHost (Aspire) in the project. | ||
| // Deprecated: Use HasImporter instead. | ||
| func (im *ImportManager) HasAppHost(ctx context.Context, projectConfig *ProjectConfig) bool { |
There was a problem hiding this comment.
[HIGH] HasAppHost behavioral change - now triggers for non-Aspire importers
HasAppHost now delegates to HasImporter, which iterates all importers including extension-provided ones. Callers like pipeline_manager.go use this to decide Aspire-specific CI behavior (InstallDotNetForAspire: props.HasAppHost). A non-Aspire extension importer returning CanImport=true would incorrectly trigger .NET SDK installation in generated CI pipelines.
Either keep HasAppHost checking only the Aspire importer by name, or rename callers to distinguish Aspire-specific behavior from generic importer presence.
| } | ||
|
|
||
| if !svc.ResourceName.Empty() { | ||
| noopMapping := func(string) string { return "" } |
There was a problem hiding this comment.
[MEDIUM] Noop env var resolver mangles resource names
The noop mapper func(string) string { return "" } replaces env var references with empty string - so rg-${AZURE_ENV_NAME} becomes rg-. This silently corrupts resource names sent to extensions. Use an identity mapper that preserves the template syntax, or send the raw unexpanded value.
| _, err = cli.RunCommandWithStdIn(ctx, stdinForInit(envName), "init") | ||
| require.NoError(t, err) | ||
|
|
||
| // Run `azd infra synth` which invokes ImportManager.GenerateAllInfrastructure |
There was a problem hiding this comment.
[LOW] Test description doesn't match what it does
Comments reference "azd infra synth" but the test runs azd show (line 110). This is a smoke test for extension startup, not infra generation. Either update the comments to match, or add an actual azd infra synth call to exercise the importer pipeline end-to-end.
Summary
Adds a new
importer-providercapability to the azd extension framework, enabling extensions to generate infrastructure (Bicep/Terraform) from project-specific definition formats.This POC enables creating extensions for scenarios like #7425, where projects want to define infrastructure in languages like C# or TypeScript instead of writing Bicep directly. The demo extension illustrates this pattern using markdown files as an analogy — where the
.mdfiles with resource definitions play the role that.csor.tsfiles would in a real implementation. The extension reads these definitions and produces the Bicep that azd provisions.How it works
azd provision: If noinfra/folder exists, the importer generates temporary Bicep at runtimeazd infra gen: The importer generates Bicep files intoinfra/(ejection)Key design decisions
infra.importer, not in the services list — services remain clean for deployable units onlyoptionsis an extension-owned map — each extension defines its own settings (path, format, etc.) with its own defaultsinfra/exists with generated files, azd uses those and skips the importerCanImporton services is preserved for existing projectsWhat is included
Core framework
Importerinterface withCanImport,Services,ProjectInfrastructure,GenerateAllInfrastructureImportManagerrefactored from hardcodedDotNetImporterto pluggable[]ImporterlistImporterRegistrysingleton for extension-registered importersImporterConfigwith extension-ownedoptionsmap inprovisioning.OptionsgRPC layer
importer.protowith bidirectionalStreamRPCImporterProviderinterface,ImporterManager,ImporterEnvelopeImporterGrpcServicewith capability verificationExternalImporteradapter implementingImporterover gRPCDemo extension
DemoImporterProviderreads.mdfiles withazd-infra-gen/v1front-matter headerazd-service-nametag)main.bicep,main.parameters.json,resources.bicepdemo-importer/(overridable viaoptions.path)Sample project (
test/functional/testdata/samples/extension-importer/)azd upto deploy to AzureDocumentation
docs/extensions/extension-custom-importers.md— authoring guideWhat is NOT included
azd initintegration: Extension importers are not invoked duringazd init. The init command uses built-inappdetectwith hardcoded language detectors. Extensions can work around this by adding their own init command (e.g.,azd my-importer init), similar toazd ai agent init. A futureproject-detectorcapability could integrate extensions intoazd initauto-detection, following the same strategy where extensions define what to detect and what to write.Closes #7425