INFO: Using directory: "1"
INFO: Using existing c-file: ./Test_Monochromator_bent.c
INFO: Using existing binary: ./Test_Monochromator_bent.out
WARNING: Ignoring invalid parameter: "Test_Monochromator_bent.instr"
INFO: ===
Simulation 'Test_Monochromator_bent' (Test_Monochromator_bent.instr): running on 16 nodes (master is 'elearn1.fysik.dtu.dk', MPI version 3.1).
*** TRACE end *** 
Detector: entry_monitor_I=0.747922 entry_monitor_ERR=0.000105772 entry_monitor_N=5e+07 "entry_monitor.dat"
double free or corruption (!prev)
double free or corruption (!prev)
# Last I/O Error: Resource temporarily unavailable
# Last I/O Error: Resource temporarily unavailable
Detector: PSD_exit_monitor_I=0.543354 PSD_exit_monitor_ERR=7.68524e-05 PSD_exit_monitor_N=5e+07 "det2d.dat"

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199252] Signal 6 detected [proc 14] SIGABRT (Abort)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_Source_finally] component Source=Source_gen() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/sources/Source_gen.comp:576] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:04 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Simulation stop (abort).

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199235] Signal 6 detected [proc 7] SIGABRT (Abort)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_Source_finally] component Source=Source_gen() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/sources/Source_gen.comp:576] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:04 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Simulation stop (abort).

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199227] Signal 6 detected [proc 0] SIGABRT (Abort)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_Source_finally] component Source=Source_gen() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/sources/Source_gen.comp:576] 0.00 % (       0.0/ 3125000.0)
corrupted size vs. prev_size
# Date:      Sun Sep 29 03:01:04 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Simulation stop (abort).
# Last I/O Error: No such file or directory
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[elearn1:2199233] *** An error occurred in MPI_Allreduce
[elearn1:2199233] *** reported by process [3560570881,6]
[elearn1:2199233] *** on communicator Test_Monochromator_bent
[elearn1:2199233] *** MPI_ERR_OTHER: known error not in list
[elearn1:2199233] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[elearn1:2199233] ***    and potentially your MPI job)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199228] Signal 15 detected [proc 1] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199229] Signal 15 detected [proc 2] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)
[elearn1.fysik.dtu.dk:2199232] pml_ucx.c:907  Error: mca_pml_ucx_send_nbr failed: -25, Connection reset by remote peer
[elearn1.fysik.dtu.dk:2199233] pml_ucx.c:907  Error: mca_pml_ucx_send_nbr failed: -25, Connection reset by remote peer
[elearn1.fysik.dtu.dk:2199241] pml_ucx.c:907  Error: mca_pml_ucx_send_nbr failed: -25, Connection reset by remote peer
# Last I/O Error: Resource temporarily unavailable
[elearn1.fysik.dtu.dk:2199255] pml_ucx.c:907  Error: mca_pml_ucx_send_nbr failed: -25, Connection reset by remote peer

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199230] Signal 15 detected [proc 3] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199231] Signal 15 detected [proc 4] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199232] Signal 15 detected [proc 5] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199233] Signal 15 detected [proc 6] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199237] Signal 15 detected [proc 8] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199239] Signal 15 detected [proc 9] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199241] Signal 15 detected [proc 10] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199243] Signal 15 detected [proc 11] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199246] Signal 15 detected [proc 12] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199249] Signal 15 detected [proc 13] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199249] Signal 11 detected [proc 13] SIGSEGV (Mem Error)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_save] component entry_monitor=PSD_monitor() SAVE [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:106] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Simulation stop (abort).

# McStas 3.x-dev - Sep. 28, 2024: [pid 2199255] Signal 15 detected [proc 15] SIGTERM (Termination)
# Simulation: Test_Monochromator_bent (Test_Monochromator_bent.instr) 
# Breakpoint: [_entry_monitor_finally] component entry_monitor=PSD_monitor() FINALLY [/home/nexmap/pkwi/McStas/mcstas/3.x-dev/monitors/PSD_monitor.comp:119] 0.00 % (       0.0/ 3125000.0)
# Date:      Sun Sep 29 03:01:05 2024
# Started:   Sun Sep 29 03:01:00 2024
# McStas 3.x-dev - Sep. 28, 2024: Finishing simulation (save results and exit)
--------------------------------------------------------------------------
mpirun noticed that process rank 14 with PID 0 on node elearn1 exited on signal 3 (Quit).
--------------------------------------------------------------------------
[elearn1.fysik.dtu.dk:2199223] 3 more processes have sent help message help-mpi-errors.txt / mpi_errors_are_fatal
[elearn1.fysik.dtu.dk:2199223] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
INFO: call to mpirun failed with Command 'mpirun -np 16 ./Test_Monochromator_bent.out --trace=0 --seed=1000 --ncount=50000000.0 --dir=1 --format=McCode --bufsiz=10000000 mono_mos=0' returned non-zero exit status 131.
Traceback (most recent call last):
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mcrun.py", line 579, in <module>
    main()
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mcrun.py", line 551, in main
    mcstas.run()  # in mccode.py
    ^^^^^^^^^^^^
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 330, in run
    return self.runMPI(args, pipe, override_mpi)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 373, in runMPI
    return Process(binpath).run(args, pipe=pipe)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 77, in run
    raise err
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/tools/Python/mcrun/mccode.py", line 73, in run
    proc = run(command, shell=True, check=True, text=True, capture_output=pipe)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nexmap/pkwi/McStas/mcstas/3.x-dev/miniconda3/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'mpirun -np 16 ./Test_Monochromator_bent.out --trace=0 --seed=1000 --ncount=50000000.0 --dir=1 --format=McCode --bufsiz=10000000 mono_mos=0' returned non-zero exit status 131.