01 · A
Problem
A whole-layout output changes dimension with bridge length and pier count. The project reformulates layout as a finite sequence of next-span decisions while removing impossible actions before they reach a solver.
All projects06 / 09
Constrained sequential decisions
Bridge Span Planning
A shared bridge-layout model with interpretable cost terms, classical baselines and masked reinforcement-learning solvers for varying bridge lengths.
- Period
- 2024 — 2026
- Focus
- Reinforcement Learning · Combinatorial Optimization · Research Engineering
- Core stack
- Python · PyTorch · Beam-DP · DQN
TL;DR
- Problem
- A whole-layout output changes dimension with bridge length and pier count. The project reformulates layout as a finite sequence of next-span decisions while removing impossible actions before they reach a solver.
- What I did
- Reformulated the finite-horizon bridge problem as an MDP/DAG and designed legal-action masks.
- Result / limit
- Established a reproducible training and inference path; runtime, cost and optimality comparisons still require a complete shared evaluation setup.
01 · B
Context
The same domain model is used by Random, cost-aware Greedy, bounded Beam-DP, tabular control and DQN, with offline training paths for HPC/Slurm.
Why it matters
Bridge layout mixes hard exclusion zones with competing, history-dependent costs. Shared constraints and interpretable baselines reveal invalid proposals and make learned policies comparable; they do not certify a bridge design.
Explore technical detailsWhat made it hard · System path · Key trade-offs
What made it hard
- 01
Bridge type, length, span catalogue and spatial resolution change the search space.
- 02
Rivers, roads and other regions can become absolute exclusions or graded costs.
- 03
The full objective is history-dependent, so beam-pruned DP is not a general exact optimum.
- 04
Illegal actions must be masked in exploration and Bellman targets.
Problem reformulation
Predicting a whole layout couples the model to bridge length
The sequential formulation keeps one stable action vocabulary: choose the next legal span or end the episode.
01coupled
Whole-layout output
Output size changes with piers and length
02biased start
Move-pier actions
Long fine-grained decision chains
03reusable
Next-span action
Stable discrete span catalogue
04variable length
End action
Episode horizon changes, output contract does not
My approach
System path
Model
Encode bridge problem, zones, legal spans and decomposed cost
Reference
Run Random, Greedy and bounded Beam-DP through one solver contract
Learn
Train tabular control and DQN with replay, target network and masks
Validate
Load versioned policies and compare on held-out problem directories
Masked decision step
Illegal spans are removed before the policy can choose them
The mask is derived from the current position, remaining length, span catalogue and exclusion zones, and is reused in exploration and Bellman targets.
01observed
State
Problem, current position and full span history
02vocabulary
Candidate actions
Discrete next spans plus end
03hard rule
Feasibility mask
Remove exclusions and impossible remainder
04shared contract
Solver choice
Greedy, Beam-DP, tabular RL or DQN
05transition
Updated state
Append span and decompose cost
Key trade-offs
Sequential reformulation
Choosing the next span keeps the action vocabulary stable while bridge length changes only the episode horizon.
DP and RL together
Bounded Beam-DP provides an interpretable strong reference; RL explores amortized inference under the same model.
Masks are correctness
Illegal actions are removed before selection and bootstrap instead of punished after execution.
04 · A
My contribution
- Reformulated the finite-horizon bridge problem as an MDP/DAG and designed legal-action masks.
- Built the shared Python solver contract and Random, Greedy, bounded Beam-DP, tabular RL and DQN paths.
- Added versioned policy artifacts, train/validation/test problem directories and regression tests.
04 · B
What was implemented
- Implemented one inspectable domain model across algorithmic and learned solvers.
- Established a reproducible training and inference path; runtime, cost and optimality comparisons still require a complete shared evaluation setup.
Lessons learned
01A learned policy is meaningful only against exactly the same constraint and cost definition.
02Fast inference is a trade-off, not evidence of a better or safer bridge layout.
What I can share
I implemented a shared model, solver families, masks, a DQN replay/target-network path and versioned artifacts. The current solver uses DQN rather than Double DQN; I leave out earlier benchmark numbers because their evaluation setup is incomplete.
Public sources