Name: slurm-sql	Distribution: openSUSE Tumbleweed
Version: 25.11.2	Vendor: openSUSE
Release: 1.2	Build date: Fri Jan 30 16:35:41 2026
Group: Productivity/Clustering/Computing	Build host: reproducible
Size: 559960	Source RPM: slurm-25.11.2-1.2.src.rpm
Packager: https://bugs.opensuse.org
Url: https://www.schedmd.com
Summary: Slurm SQL support

Contains interfaces to MySQL for use by SLURM.

Provides

Requires

License

SUSE-GPL-2.0-with-openssl-exception

Changelog

* Fri Jan 30 2026 Christian Goll <cgoll@suse.com>
- Removed the testsuite as it's not part of normal release any more
- Removed Fix-test7.2-to-find-libpmix-under-lib64-as-well.patch
- Update to version 25.11.2.1
* slurmstepd - Revert regression that would apply job environment to
container runtime invocation.
* Fix issue where reservations may start while required GRES resources are
still being used by jobs.
* Fix slurmctld segfault when using --consolidate-segments.
* Expose slurm.CONSOLIDATE_SEGMENTS flag in lua.
* Expose the job record's segment_size in lua.
* job_submit/lua - Expose the job_desc's segment_size in lua.
* Prevent PMIx 5.0.8 and 5.0.9 clients from hanging when connecting to the
PMIx server.
* Clarify warning when BPF tokens are not supported.
* slurmctld - Ensure we close already accepted conn before RPC flush check
* slurmctld - Fix rpc_queue feature causing statesave corruption while
shutdown
* slurmctld - Ensure backfill has finished before saving state.
* slurmctld - Ensure main scheduler has finished before saving state.
* slurmctld - Fix error message while shutting down and state cannot be
saved.
* Fix slurmctld double free that occurs when purging array jobs from memory
only when using the topology/block plugin.
* Fix steps being rejected inside a batch job when using --cpus-per-task and
- -mem-per-cpu, and the job was submitted to multiple partitions, but not
all of them had the same MaxMemPerCPU limit in place.
* slurmctld - Fix crash after failed reconfiguration while running jobs and
priority/multifactor enabled.
* slurmctld - Fix jobs' QOS/association usage leading to potential underflow
errors after a failed reconfiguration attempt.
* Guess NodeName with gethostname instead of gethostname_short
* Fix allowing job submissions when EnforcePartLimits=NO and the requested
minimum number of nodes exceeds the total nodes in the specified
partition(s).
* Fix double unlock issue in _slurm_rpc_job_sbcast_cred()
* srun - fix bug where some input/output/error filename format identifiers
were not expanded.
* Fix detecting restricted cores with SlurmdSpecOverride in nodes with more
than one socket.
* slurmctld/slurmdbd - Prevent segfaulting if a persistent connection closes
right before reconfiguring or shutting down.
* Fix average calculation in latency timers to show more accurate timing
logs.
- Update to version 25.11.1.1
* data_parser/v0.0.41 - Prevent memory leaks when freeing parsed lists.
* data_parser/v0.0.42 - Prevent memory leaks when freeing parsed lists.
* data_parser/v0.0.43 - Prevent memory leaks when freeing parsed lists.
* data_parser/v0.0.44 - Prevent memory leaks when freeing parsed lists.
* slurmctld - Prevent a fatal when min_exempt_priority is not the last
option listed in PreemptParameters.
* Updating a job's qos will always replace the previous timelimit with the
new qos' timelimit, unless another time limit is explicitly specified in
the update command.
* When debugflags=script is set in slurm.conf, Lua runtime error message
will be logged with backtrace.
* slurmctld - Prevent memory corruption when fanning out messages to the
slurmds if TreeWidth is more then or equal to 46341 and the number of
nodes in the cluster is more then or equal to (TreeWidth + 1).
* When GrpTRES and MaxTRESPU are set on different QOSes and both QOSes are
applied to a job, ensure that both limits are honored.
* Fix issue where a cli command or process could get stuck indefinitely when
trying to retrieve a slurm.conf from slurmctld.
* Fix slurmctld potential deadlock when trying to schedule jobs starting
many years in the future. Slurm only supports one year time limits.
* Fix pam_slurm_adopt when using namespace/linux plugin.
* topology/tree - Prevent overflow error when calculating fanout depth.
* The state string for nodes in the MIXED+FAIL state will now appear as
"FAILING" rather than just "FAIL", similar to what is already done for
nodes in the ALLOCATED+FAIL state.
* slurmctld - Prevent a divide by zero crash by fataling if the following
SlurmctldParameters have a value of less than or equal to 0:
rl_table_size, rl_bucket_size, rl_refill_rate, and rl_refill_period.
* Fix missing updates to reservation TRES and accounting when node(s)
replaced due to REPLACE or REPLACE_DOWN flags.
* slurmctld - Cancel interactive job if prolog RPC never reaches its
receiver.
* slurmctld - Cancel interactive jobs that never ran the prolog in the purge
jobs logic.
* Fix accounting for memory on steps without pids, like the extern step,
which caused them to be killed if OvermemoryKill was set.
* NO_NORMAL_ALL will only be printed if all NO_NORMAL_* flags are set.
* slurmctld - Prevent the controller from believing it has a job's
federation cluster lock when it does not.
* Fix jobs incorrectly stuck waiting for resources when launched with
specific client flag combinations containing "--hint=nomultithread".
* Fix allocated licenses still showing after removing all allocated
licenses.
* accounting_storage/mysql - Disallow creating users if requested user list
is empty or usernames are empty strings.
* slurmrestd - Revert tagging .script field as deprecated in 'POST
/slurm/v0.0.42/job/submit'.
* slurmrestd - Revert tagging .script field as deprecated in 'POST
/slurm/v0.0.43/job/submit'.
* slurmrestd - Revert tagging .script field as deprecated in 'POST
/slurm/v0.0.44/job/submit'.
* slurmrestd - Revert regression that changed the error from "Authentication
failure" to "Authentication does not apply to request" when a HTTP request
lacks any authentication credentials.
* When a job requests multiple partitions and cannot run in one of them due
to topology, allow the main scheduler to evaluate jobs in the other
requested partitions.
* slurmctld - Acquire the node write lock instead of the node read lock when
querying 'GET /metrics/nodes' and 'GET /metrics/partitions' endpoints.
* slurmctld - Fixed segfault when running configless and a malformed
REQUEST_CONFIG RPC is received.
* Remove error output for missing optional spank plugin.
* slurmctld - when unable to schedule a job with preferred node features,
don't exclude the partition from further scheduling attempts in the same
iteration.
* Fix issue with RestrictedCoresPerGPU with shared gres.
* Fix rpmbuild --with libcurl option.
* Add new JobAcctGatherParams=no_file_cache to change how memory usage (RSS)
is reported when using cgroup/v2. With this flag set we will subtract
active_file and inactive_file from the value reported in memory.current to
avoid counting the file cache. memory.peak will then not be used to get
the MaxRSS and getting memory spikes will depend on the
JobAcctGatherFrequency parameter.
* namespace/linux - fix bug that could leave defunct processes in the jobs
namespace.
* namespace/linux - kill and reap the namespace process during job teardown.
* namespace/linux - Fix issue with user_ns_script that may result in STDIN
closing, which may result in 'Unable to receive "ok ack"' error on
slurmstepd or other undefined behavior.
* Fix error reading /proc/0/* when calling the api outside the step
namespace.
* slurmctld - Fixed segfault when using newly added remote licenses.
* Fix SIGCHLD not being sent to tasks.
* bitmap2node_name() is not cleaned up properly when reservation logging is
enabled.
* Fix issue with jobs running on slurmd's with version 25.05.x or older
getting aborted when slurmd re-registers with slurmctld.
* Fix memory leak on slurmctld for jobs that use --exclusive=topo
* Prevent jobs that cannot fit in the reservation's time limit from being
attracted to a magnetic reservation.
* Fix slurmstepd segfault for older versioned batch jobs (25.05 and older)
submitted without using -o/--output on submission.
- Update to version 25.11.0.1
* namespace/linux - move directory creation for bind mounts to before the
init script is called.
* namespace/linux - add SLURM_JOB_MEM to script environments when able.
* Fix an error when printing sdiag rpc stats in json format when hostlists
strings are too long.
* Add --no-trunc argument to sdiag. That will output long hostlists that
default to being truncated to 80 characters.
* Add infinite (-1) layer support to HRes mode 3.
* Fix ESLURM_RETRY_EVAL handling in common_topo_choose_nodes().
* Fix HRes MODE_3 when using with --gpus.
* Fix enforcing of MODE_3 with --distribution=arbitrary.
* slurmrestd - Fix regression that caused rejected HTTP requests to not
include an descriptive error message.
* slurmrestd - Fix regression that caused requests for unknown or
unsupported URL paths to not include a descriptive error.
- Update to version 25.11.0rc2
* slurmd - Avoid segfault during startup when /sys/ is not mounted correctly
and gpu/nvidia plugin is configured.
* Fix compilation when building with SLURMSTEPD_MEMCHECK == 1.
* slurmresd - Catch use-after-free bugs for closed connections.
* Fix building with libyaml in a non-standard location.
* Suppress false error messages when system.slice cgroup is not present and
EnableControllers is set.
* Adjust the OOM score of slurmstepd processes from -1000 to -999 to make
them killable. This change addresses cases where slurmstepd consumes more
memory than what is allocated to the job's cgroup. Previously, being
unkillable could cause the process to get stuck during memory allocation,
fully occupy a CPU core, and flood the kernel logs.
* slurmrestd - Prevent crash when an empty request is submitted to 'POST
slurm/*/job/submit' endpoints.
* Allow tres-bind on steps with only one task.
* Fix INVALID nodes going DRAIN after a slurmd restart without any gres.conf
modification.
* slurmrestd - Prevent potential crash when using the 'POST
/slurmdb/*/accounts_association' endpoints.
* data_parser/v0.0.44 - Make field 'association_condition' required to
prevent a slurmrestd crash when the field is not provided. This affects
the following endpoints: 'POST /slurmdb/v0.0.44/accounts_association/'
* Add no_tag as a possible parameter to environment SLURMRESTD_YAML or
environment SLURM_YAML to disable dumping YAML datatype !!tags for CLI
commands supporting --yaml output.
* Fix non-fatal "No such file or directory" build errors when building on
* EL systems.
* Slurmd will now unload gpu plugin after configuration is over, unless
acct_gather_energy/gpu is set.
* Improve stepd termination messages for non-zero return codes.
* Fix double-deducting licenses when recovering COMPLETING jobs from state.
* Remove AccountingStorageUser option from slurm.conf.
* data_parser/v0.0.44 - Add five reservation flags that were missing from
the RESERVATION_FLAGS array. This affects scontrol show reservation
- -{json|yaml} and the following REST API endpoints:
+ 'GET /slurm/v0.0.44/reservation/{reservation_name}'
+ 'GET /slurm/v0.0.44/reservations'
+ 'POST /slurm/v0.0.44/reservation'
+ 'POST /slurm/v0.0.44/reservations'
* Fix Slurm components that depend on libjson-c not setting RUNPATH when
requested, which could cause them to fail at runtime.
* Fix Slurm components that depend on libyaml not setting RUNPATH when
requested, which could cause them to fail at runtime.
* slurm.spec - Remove duplicated --with-freeipmi
* slurmrestd - Fix crash for /reservations endpoint when a valid
reservation_desc_msg body with a specified partition but no node_list
string array is submitted.
* Fix memory leak in configless mode when resolving the slurmctld address.
* slurmrestd - Avoid segfault with -d list args when no data_parser plugins
can be read by process which requires removal of plugins from
LD_LIBRARY_PATH or RPATH or some other administrative action as they are
always installed with Slurm.
* Change pam_slurm_adopt install location for the Debian package to the
multiarch location.
* Support conversion of JSON/YAML dictionary/object to list/array where
automatic type inferencing is supported.
* slurmrestd - Avoid giving database connection hex address in warnings when
a slurmdbd query "found nothing". All GET /slurmdb/v0.0.*/* endpoints are
affected by this change.
* slurmrestd - Avoid giving database connection hex address in warnings when
a slurmdbd query "reports nothing changed". All GET /slurmdb/v0.0.*/*
endpoints are affected by this change.
* slurmrestd - Avoid giving database connection hex address in warnings when
a slurmdbd query "failed". All GET /slurmdb/v0.0.*/* endpoints are
affected by this change.
* slurmrestd - Avoid giving database connection hex address in errors when a
slurmdbd query "failed". All GET /slurmdb/v0.0.*/* endpoints are affected
by this change.
* slurmrestd - Avoid giving database connection hex address in errors when a
slurmdbd query "failed". All POST /slurmdb/v0.0.*/* endpoints are affected
by this change.
* slurmrestd - Avoid giving database connection hex address in errors when a
slurmdbd query "failed" to commit changes. All POST /slurmdb/v0.0.*/*
endpoints are affected by this change.
* Add log_user() to be callable by all Lua scripts.
* slurmctld - Relock the controller's pidfile on a reconfigure.
* Add --with cgroupv2 option to slurm.spec to assist building rpms on
systems that require cgroupv2 support.
* Pass existing cluster id from slurmctld to slurmdbd when registering the
cluster with accounting for the first time
* scontrol - Improve error message when attempting to perform an invalid
state update on a node, e.g. RESUME a node that is currently in a state
from which resuming is not possible.
* Set exit_code when "scontrol listjobs/liststeps" does not find jobs/steps.
* Fix rejecting some valid jobs that requested --sockets-per-node > 1 and
- -gres-flags=enforce-binding.
* For some jobs that use --gres-flags=enforce-binding, allocate the lowest
numbered socket first. The previous behavior allocated the highest
numbered socket first.
* Fix some situations where --gres-flags=enforce-binding was not respected
for jobs that request --gpus-per-task and multiple nodes.
* sacctmgr - Catch and reject attempts to pass invalid account flags=
arguments.
* Avoid returning an "INVALID" Accounting flag in JSON or YAML output which
was never valid flag.
* sacctmgr - Catch and reject attempts to pass invalid Cluster flags=
arguments.
* Avoid returning an "INVALID" Cluster flag in JSON or YAML output which was
never a valid flag.
* Avoid returning an "INVALID" Association flag in JSON or YAML output which
was never a valid flag.
* sacctmgr - Catch and reject attempts to pass invalid QOS flags.
* Avoid returning an "INVALID" QOS flag in JSON or YAML output which was
never a valid flag.
* slurmdbd.conf can have permissions of 640 or 600.
* Avoid possible race condition that could cause a hang if process should
shutdown before all conmgr worker threads have started.
* Add HttpParserType parameter to slurm.conf
* Add UrlParserType parameter to slurm.conf
* Add http_parser/libhttp_parser plugin.
* slurmrestd - Switch to using http_parser/libhttp_parser plugin for http
processing.
* topology/block - Add new TopologyParam=BlockAsNodeRank option to reorder
nodes based on block layout. This can be useful if the naming convention
for the nodes does not natually map to the network topology.
* Fix memory leak in acct_gather_profile_influxdb.c.
* Allow --segment to be bigger than base block size.
* sacctmgr - Allow an operator to alter the allocated TRES of a job.
* sacctmgr - add new option to set fixed runaway jobs as FAILED or
COMPLETED.
* Job option --hint is now mutually exclusive with both --cores-per-socket
and --sockets-per-node.
* Add HealthCheckNodeState=START_ONLY option.
* Add a new CliFilterParameters=cli_filter_lua_path= to slurm.conf, enabling
the configuration of an absolute path to the cli_filter.lua script when
cli_filter/lua is configured.
* Allow removing a user's default association when AllowNoDefAcct=yes.
* scontrol - Fix "KillOnInvalidDependent" typo in show jobs output.
* scontrol - Use whitespace to separate all key-value pairs in show jobs
output. Previously, commas were used between job flags.
* scontrol - Fix GresAllowTaskSharing not always appearing in show jobs
output.
* slurmrestd - Log URLs that fail to parse under the debugflag=data instead
as errors or debug5 as invalid user provided is not directly an error of
the slurmrestd daemon itself.
* Fix use-cases incorrectly rejecting job requests when
MaxCPUsPer[Socket|Node] applied and CPUSpecList/CoreSpecCount configured.
* slurmrestd - Specify listening on UNIX socket by giving the "unix://"
prefix instead of "unix:" prefix for the URL scheme.
* Don't subtract 128 from sacct returned exit codes for the codes above 128.
* Fix issue resolving socket max segment size (MSS) via kernel to more
accurately size buffered communications.
* mpi/pmix - Restore the ability to follow symlinks on PMIx cli/lib temporal
directory creation when considered trusted (i.e. SlurmdSpoolDir, TmpFS,
PMIxCliTmpDirBase). This ability was lost in Slurm 23.02.6.
* Invalid use of -= or += aborts sacctmgr commands with an error.
* Add SLURM_JOB_SEGMENT_SIZE environment variable to salloc and srun job
environments when --segment is set.
* Remove the trailing space after AdminComment, SystemComment, Comment, and
Extra fields in scontrol show jobs output.
* slurmdbd - Make sure locks are appropriate when modifying or removing
associations while filtering by QOS.
* defaultqos is not set when loaded from dump file
* Add display for default qos when listing users and not specifying
"withassoc"
* slurmctld - Always close rejected incoming RPC connections to avoid
reading another incoming RPC request.
* slurmctld - Log runtime errors from global section of Lua scripts during
loading.
* Allow gpu shared gres to use RestrictedCoresPerGPU.
* slurmrestd - Fix memory leak that happened when submitting a request body
containing the "warnings", "errors", or "meta" field. This affects the
following endpoints: 'POST /slurmdb/v0.0.4*/qos'
* slurmrestd - Prevent triggering a fatal abort when parsing a non-empty
group id string. This affects all endpoints with request bodies containing
openapi_meta_client group field. It also affects the following endpoints:
+ 'GET /slurmdb/v0.0.4*/jobs'
+ 'POST /slurm/v0.0.4*/job/submit'
+ 'POST /slurm/v0.0.4*/job/{job_id}'
+ 'POST /slurm/v0.0.4*/job/allocate'
* Add the ability to gate access to a reservation by QOS.
* Add the ability to gate access to a reservation by Partition.
* scontrol - Add "SubmitLine" field to show job subcommand's output.
* slurmctld - Prevent an invalid read and a possible crash by rejecting any
arbitrary distribution jobs that do not specify a task count equal to the
number of node names in their node list. This does not affect srun,
salloc, or sbatch if -n is not used since they set the default task count.
* slurmrestd - Load slurmctld and slurmdbd OpenAPI plugins by default when
slurmdbd accounting is configured or only the slurmctld OpenAPI plugin
otherwise.
* slurmrestd - Improve error response message when URL is not found or not
supported.
* Fix race condition that could result in a syslog message being lost right
immediately before executing a script.
* Fix race condition that could result in log or scheduling log message
being lost or misdirected immediately before executing a script.
* Add support for "+=" and "-=" syntax when modifying TRES options via
"sacctmgr modify". Supported for QoS, User, Account and Cluster entities.
* slurmctld - Avoid falsely setting partially closed sockets as stale.
* slurmctld - Check if persistent connections have gone stale before
attempting to respond to RPC requests.
* slurmctld - Enforce stale connection check on RPCs with asynchronous I/O.
* TaskProlog and TaskEpilog no longer have to be fully qualified pathnames.
* When configless is enabled, TaskProlog and TaskEpilog scripts can now be
pushed to slurmds if the script files are in the same directory as
slurm.conf.
* When replacing nodes due to REPLACE flag, do not pull nodes from MAINT
reservations.
* Add a new --spread-segments option to salloc/sbatch/srun.
* tls/s2n - Fix PMIx server not working.
* slurmrestd - Add several more verbose errors for rejected requests in
slurmrestd's logs instead of "Unspecified error".
* slurmrestd - Add specific error when Content-Length header is too large
for any HTTP request.
* slurmrestd - Fix a bug in the "GET /slurm/v0.0.4*/node/{node_name}"
endpoint where the node's partitions field would be incorrectly populated.
* Log Lua backtrace on Lua script failures while loading scripts when
debugflags=script and at at least log level debug when Slurm is linked to
all Lua releases after version 5.4.6.
* Add 'MaxPurgeLimit' to 'slurmdbd.conf', replacing constant 50,000 limit.
* slurmrestd - Avoid logging redundant "Unspecified error" after failing to
load TLS certificates during startup.
* Fix gres enforce-binding not allowing multi-socket tasks with gres-per-
task requests.
* Add limited support for the conjunction of "--segment" and "--nodelist"
("-w") for topology/block.
* squeue - Add new --running-over and --running-under options, used to
filter jobs based on their runtime.
* Fix the 'is-busy' check for a node shared between partitions.
* Fix incorrect use of bind() on BSD platforms.
* Fix handling of strigger --fail/-F event.
* slurmctld - Check connections can read and write when testing if
connection has gone stale for incoming RPCs.
* Fix power save scripts that close stderr and stdout from being killed at
shutdown after 10 second timeout.
* slurmctld - Avoid edge cases causing file descriptors to never close when
an incoming RPC connection fails.
* slurmctld - Prevent the gang scheduler from hitting a segfault on restart,
reconfigure, or when using 'scontrol delete partition' if all partitions
are removed.
* data_parser/v0.0.44 - Added missing node state flag BLOCKED to parsable
node states. This affects 'scontrol show nodes --{json|yaml}', 'sinfo
- -{json|yaml}', and the following REST API endpoints:
+ 'GET /slurm/v0.0.44/nodes'
+ 'GET /slurm/v0.0.44/node/{node_name}'
* scontrol - Fix ping showing the wrong hostname when using the --cluster
option. The correct cluster name will now be displayed in the output when
using --cluster.
* Job stderr string will be blank in data_parser output, scontrol and sview
unless it has been explicitly set in the job. Users must assume that
stderr will default to the stdout path in that case.
* Use the node_name instead of hostname in file name creation.
* Set CgroupPlugin=disabled by default when built without cgroup support.
* CgroupPlugin=Autodetect will always return "disabled" when Slurm is built
without cgroup support.
* Do not build cgroup plugins on OpenBSD.
* mpi/pmix - The PMIx server will no longer store the node's HWLOC topology
in the job-level information under the PMIX_LOCAL_TOPO key by default. The
new PMIxShareServerTopology mpi.conf option can be used to share the
topology in PMIx v4+.
* slurmctld/slurmd - Fatal instead of crashing if no blocks or switches are
defined in topology.conf when using topology/tree or topology/block.
* slurmrestd - Honor HTTP "Connection: Keep-Alive" header to enable
persistent connections for HTTP/1.1 connections.
* slurmrestd - Reject HTTP Keep-Alive headers to enable persistent
connections for HTTP/0.9 connections.
* slurmrestd - Ignore HTTP "Connection: Keep-Alive" headers for HTTP/1.1
connections as they are persistent connections without "Connection: Close"
headers by default.
* slurmrestd - Ignore HTTP "Keep-Alive: *" headers as no valid parameters
were ever defined instead of logging a warning.
* slurmrestd - Replies to unparsable or unsupported requests will be
HTTP/1.1 formatted.
* Reimplement fs/lustre stats gathering to ensure it only accounts for usage
since the beginning of the job, as it was previously incorrectly reporting
usage since last stats reset. Also add the ability to gracefully handle
reset/overflow of the Lustre counters.
* Remove node_features/knl_generic plugin.
* Limit conmgr_threads to 256 for all configuration parameters.
* slurmctld - Avoid potentially logging invalid address for info level
logging of "slurmctld_req: received opcode %s from %pA uid %u".
* slurmrestd - Allow externally signed TLS certificates to be used for
https:// sockets. Certificate no longer needs to be signed by the
ca_cert_file that is used for internal Slurm TLS communications.
* Add a new 'reconfig_on_restart' to 'SlurmctldParameters' making a restart
of slurmctld (a process restart that was not triggered by "scontrol
reconfigure") trigger a reconfigure of all slurmd and sackd daemons.
* Skip attempting to resolve reserved user "nobody" (uid:99) using
getpwduid_r().
* Skip attempting to resolve reserved group "nobody" (uid:99) using
getgrgid_r().
* Always resolve group id 99 as group "nobody".
* Always resolve user id 99 as user "nobody".
* slurmctld,slurmd,slurmstepd,slurmdbd,slurmrestd,scrun: Avoid SEGFAULT
during shutdown due to race condition with receiving POSIX signal when
debugflags=conmgr.
* Avoid race condition when handling a TLS wrapped connection when the TLS
plugin is not loaded.
* switch/nvidia_imex - Fix channel count being one less than the actual
imex_channel_count configuration.
* switch/nvidia_imex - Add --network=unique-channel-per-segment option to
allocate a unique IMEX channel per segment inside a job when using the
topology/block plugin.
* switch/nvidia_imex - When the allocated number of IMEX channels hits
imex_channel_count, new jobs will start pending instead of running without
creating any IMEX channels.
* slurmrestd - Add environment variables SLURMRESTD_DEBUG_SYSLOG and
SLURMRESTD_DEBUG_STDERR to allow control of logging targets.
* slurmrestd - Add support for the SLURM_DEBUG environment variable.
* Add a new --consolidate-segments option to salloc/sbatch/srun.
* openapi/util - Add openapi/util plugin for request not requiring a
connection to the slurmctld or slurmdbd.
* slurmrestd - Load the openapi/util plugin by default.
* slurmctld,sackd,slurmdbd,slurmrestd: Limit the automatically detected
conmgr thread count to 32. Avoid having too many threads on higher core
count hosts.
* slurmd - Change default to SlurmdParameters=conmgr_threads=6.
* scrun - Change default conmgr thread count to 3.
* slurmstepd - Change default conmgr_threads=4 when
SlurmdParameters=conmgr_threads=# is not defined.
* slurmctld,slurmd,slurmstepd,sackd,slurmdbd,slurmrestd: Log warnings when
configured conmgr thread counts are outside of suggested range based on
detected CPU counts.
* slurmd - Add --parameters option to append additional options to the
SlurmdParameters option from slurm.conf. Can also be used with 'slurmd -C'
to print alternate cpu topologies. (E.g., 'slurmd -C --parameters
l3cache_as_socket'.)
* Add per-node Parameters option, allowing node-specific SlurmdParameters.
* Manually updating a node's state to IDLE with 'scontrol update nodename'
is no longer allowed if the node is in state REBOOT_ISSUED. This is to
prevent the REBBOOT_ISSUED state from not automatically clearing once the
reboot finishes.
* openapi/util - Add endpoints to convert hostlist expressions to an array
of hostnames and vice versa:
+ 'POST /util/v0.0.44/hostnames'
+ 'POST /util/v0.0.44/hostlist'
* slurmrestd - Return HTTP_STATUS_CODE_ERROR_UNPROCESSABLE_CONTENT if
querying an endpoint results in ESLURM_DATA_PATH_NOT_FOUND error.
* data_parser/v0.0.44 - Avoid hitting an xassert by return a DATA_TYPE_LIST
on error when dumping a HOSTLIST_STRING without complex mode instead of
returning DATA_TYPE_NULL.
* slurmdbd - Allow slurmdbd.conf to be owned by either SlurmUser or root
instead of just SlurmUser.
* Add new DisableArchiveCommands parameter to slurmdbd.conf to disable
"sacctmgr archive" commands.
* Prevent jobs submitted to multiple partitions from being killed when
removing a partition from slurm.conf and restarting or reconfiguring the
slurmctld. Pending jobs will remain pending with the remaining partitions.
If a job's running partition, or all its partitions, are removed, the job
will get killed with JobState=NODE_FAIL and Reason=PartitionDown. Jobs
killed due to removing partitions from slurm.conf cannot be requeued.
* slurmctld - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* slurmscriptd - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* slurmstepd - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* slurmrestd - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* Allow external jobs to be submitted without requiring the current working
directory be set.
* Fix srun ntasks calculation when nodes are requested using a min-max range
and --ntasks is not used.
* Add CgroupSlice option to cgroup.conf to allow changing the default
"system.slice" slice when starting slurmd and slurmstepd. If slurmd is
started with a unit file, modifying the unit and adding Slice= into the
[Service] section is needed. After making this change in cgroup.conf, the
old slurmstepd scopes need to be terminated before starting slurmd again.
* Reject invalid dependencies with more than one dependency separator type.
* Reject dependency specifications with trailing garbage as invalid syntax.
* Mark afterok/afternotok dependencies on a non-existent remote job id as
failed. Previously the dependency would remain unfulfilled indefinitely.
* Federated clusters update every registered remote dependency, including
duplicates. This prevents jobs from pending forever due to some
dependencies never being fulfilled.
* Fix aftercorr dependency on remote jobs in a federation.
* srun - Fix not accepting incoming TTY connection from slurmstepd due to
race condition with POSIX signals during step startup.
* Add MetricsType option to slurm.conf.
* Add slurmdb endpoints to update job database:
+ 'POST /slurmdb/v0.0.44/job/{jobid}'
+ 'POST /slurmdb/v0.0.44/jobs'
* Add CommunicationParameters=disable_http to slurm.conf.
* slurmctld - Load http_parser, url_parser and TLS plugins by default when
CommunicationParameters=disable_http is not set in slurm.conf.
* slurmctld,slurmd - Add the following HTTP endpoints:
+ GET /
+ GET /livez
+ GET /readyz
+ GET /healthz
* acct_gather_profile/influxdb - Add new acct_gather.conf option of
ProfileInfluxDBFrequency to configure the frequency at which data is sent
to InfluxDB (or disable buffering entirely if set to 0). Defaults to 30s.
* Fix slurmdbd error triggered by "sreport user topusage" when trying to get
data from monthly usage tables.
* Fix systemd potentially losing track of the daemon PIDs.
* Fix fs_factor being 0 in the rest API when using
PriorityFalgs=NO_FAIR_TREE.
* slurmctld - Enforce stale connection checks for rpc_queue enabled
messages.
* switch/nvidia_imex - IMEX channels are now accessible in the batch step
and interactive step.
* Validate QOS configuration even if AccountingStorageEnforce QOS is not
set.
* Validate AllowQOS, DenyQOS have valid QOS's listed.
* Change schedule_cycle_sum to uint64_t to prevent overflows which will
otherwise occur after 2^32 microseconds (71.6 minutes).
* scontrol - Fix regression where "scontrol update jobid= qos=" was not
considered a valid command.
* When using CR_CPU, prevent counting more cpus against a step then was
allocated.
* slurmd - Load http_parser, url_parser and TLS plugins by default when
CommunicationParameters=disable_http is not set in slurm.conf.
* sackd - Add --jwks-file and --key-file options to specify a non-default
absolute path to .jwks and .key authentication key(s) file.
* sinfo --state/-t flag now outputs a more helpful error message when passed
an invalid node state.
* When using CR_CPU, prevent counting more cpus against a step then was
allocated.
* Fix CPU over-allocation with enforce-binding when ntasks is specified.
* data_parser/v0.0.44 - Add nodes field to slurm_license_info_t parser.
* Add a new mode of operation to HRes (MODE_3). In this mode, resources are
allocated on a per-node basis, with the total consumption being calculated
from the most granular layer and summed up into the successively higher
layers.
* scontrol - return error when attempting to add node to a non-existent
topology block.
* Allow cloud nodes to dynamically set topology at registration with slurmd
- -conf="topology=...".
* Add HTTP endpoints to expose Slurm metrics:
+ GET /metrics/jobs
+ GET /metrics/nodes
+ GET /metrics/partitions
+ GET /metrics/scheduler
+ GET /metrics/user-accts
* switch/hpe_slingshot - Reduce sharing of RGIDs on multi-NIC connected
systems. Default to distributing RGIDs evenly across all available NICs.
* switch/hpe_slingshot - Add nic_distribution_count option to
SwitchParameters.
* switch/hpe_slingshot - Add nic_distribution_count parameter to the
- -network job and step option.
* slurmstepd - Prevent the slurmstepd from segfaulting if the
switch/hpe_slingshot plugin is enabled and SwitchParameters is not
specified.
* openapi/slurmctld - Fix 'DELETE /slurm/v0.0.XX/node/{node_name}' error
message so that it says an issue occurred while deleting instead of
updating.
* Fix segfault with multi-node gres with PrologFlags=RunInJob.
* accounting_storage/mysql - Improve performance for 'sacct -T/--truncate'
by skipping suspend table queries for unsuspended jobs.
* Change job_container/tmpfs plugin to be named namespace/tmpfs.
* Rename DebugFlags=JobContainer to DebugFlags=Namespace
* Add new namespace/linux plugin.
* slurmctld - Close connection earlier when reply on RPC enqueuing fails.
* Add SlurmctldParameters=enable_async_reply to slurm.conf.
* Add new "scancel --admin-comment" to allow admins to set the AdminComment
while canceling jobs.
* Add support for database password rotation in slurmdbd through new
StoragePassScript option.
* Add support for database password rotation for jobcomp/mysql through new
JobCompPassScript option.
* Add interactive_step_set_cpu_freq parameter to the list of
LaunchParameters in slurm.conf. This allows for the default cpu freq
governor to be set on interactive steps.
- Update to version 25.11.0rc1
* slurmd - Avoid segfault during startup when /sys/ is not mounted correctly
and gpu/nvidia plugin is configured.
* Fix compilation when building with SLURMSTEPD_MEMCHECK == 1.
* slurmresd - Catch use-after-free bugs for closed connections.
* Fix building with libyaml in a non-standard location.
* Suppress false error messages when system.slice cgroup is not present and
EnableControllers is set.
* Adjust the OOM score of slurmstepd processes from -1000 to -999 to make
them killable. This change addresses cases where slurmstepd consumes more
memory than what is allocated to the job's cgroup. Previously, being
unkillable could cause the process to get stuck during memory allocation,
fully occupy a CPU core, and flood the kernel logs.
* slurmrestd - Prevent crash when an empty request is submitted to 'POST
slurm/*/job/submit' endpoints.
* Allow tres-bind on steps with only one task.
* Fix INVALID nodes going DRAIN after a slurmd restart without any gres.conf
modification.
* slurmrestd - Prevent potential crash when using the 'POST
/slurmdb/*/accounts_association' endpoints.
* data_parser/v0.0.44 - Make field 'association_condition' required to
prevent a slurmrestd crash when the field is not provided. This affects
the following endpoints: 'POST /slurmdb/v0.0.44/accounts_association/'
* Add no_tag as a possible parameter to environment SLURMRESTD_YAML or
environment SLURM_YAML to disable dumping YAML datatype !!tags for CLI
commands supporting --yaml output.
* Fix non-fatal "No such file or directory" build errors when building on
* EL systems.
* Slurmd will now unload gpu plugin after configuration is over, unless
acct_gather_energy/gpu is set.
* Improve stepd termination messages for non-zero return codes.
* Fix double-deducting licenses when recovering COMPLETING jobs from state.
* Remove AccountingStorageUser option from slurm.conf.
* data_parser/v0.0.44 - Add five reservation flags that were missing from
the RESERVATION_FLAGS array. This affects scontrol show reservation
- -{json|yaml} and the following REST API endpoints:
+ 'GET /slurm/v0.0.44/reservation/{reservation_name}'
+ 'GET /slurm/v0.0.44/reservations'
+ 'POST /slurm/v0.0.44/reservation'
+ 'POST /slurm/v0.0.44/reservations'
* Fix Slurm components that depend on libjson-c not setting RUNPATH when
requested, which could cause them to fail at runtime.
* Fix Slurm components that depend on libyaml not setting RUNPATH when
requested, which could cause them to fail at runtime.
* slurm.spec - Remove duplicated --with-freeipmi
* slurmrestd - Fix crash for /reservations endpoint when a valid
reservation_desc_msg body with a specified partition but no node_list
string array is submitted.
* Fix memory leak in configless mode when resolving the slurmctld address.
* slurmrestd - Avoid segfault with -d list args when no data_parser plugins
can be read by process which requires removal of plugins from
LD_LIBRARY_PATH or RPATH or some other administrative action as they are
always installed with Slurm.
* Change pam_slurm_adopt install location for the Debian package to the
multiarch location.
* Support conversion of JSON/YAML dictionary/object to list/array where
automatic type inferencing is supported.
* slurmrestd - Avoid giving database connection hex address in warnings when
a slurmdbd query "found nothing". All GET /slurmdb/v0.0.*/* endpoints are
affected by this change.
* slurmrestd - Avoid giving database connection hex address in warnings when
a slurmdbd query "reports nothing changed". All GET /slurmdb/v0.0.*/*
endpoints are affected by this change.
* slurmrestd - Avoid giving database connection hex address in warnings when
a slurmdbd query "failed". All GET /slurmdb/v0.0.*/* endpoints are
affected by this change.
* slurmrestd - Avoid giving database connection hex address in errors when a
slurmdbd query "failed". All GET /slurmdb/v0.0.*/* endpoints are affected
by this change.
* slurmrestd - Avoid giving database connection hex address in errors when a
slurmdbd query "failed". All POST /slurmdb/v0.0.*/* endpoints are affected
by this change.
* slurmrestd - Avoid giving database connection hex address in errors when a
slurmdbd query "failed" to commit changes. All POST /slurmdb/v0.0.*/*
endpoints are affected by this change.
* Add log_user() to be callable by all Lua scripts.
* slurmctld - Relock the controller's pidfile on a reconfigure.
* Add --with cgroupv2 option to slurm.spec to assist building rpms on
systems that require cgroupv2 support.
* Pass existing cluster id from slurmctld to slurmdbd when registering the
cluster with accounting for the first time
* scontrol - Improve error message when attempting to perform an invalid
state update on a node, e.g. RESUME a node that is currently in a state
from which resuming is not possible.
* Set exit_code when "scontrol listjobs/liststeps" does not find jobs/steps.
* Fix rejecting some valid jobs that requested --sockets-per-node > 1 and
- -gres-flags=enforce-binding.
* For some jobs that use --gres-flags=enforce-binding, allocate the lowest
numbered socket first. The previous behavior allocated the highest
numbered socket first.
* Fix some situations where --gres-flags=enforce-binding was not respected
for jobs that request --gpus-per-task and multiple nodes.
* sacctmgr - Catch and reject attempts to pass invalid account flags=
arguments.
* Avoid returning an "INVALID" Accounting flag in JSON or YAML output which
was never valid flag.
* sacctmgr - Catch and reject attempts to pass invalid Cluster flags=
arguments.
* Avoid returning an "INVALID" Cluster flag in JSON or YAML output which was
never a valid flag.
* Avoid returning an "INVALID" Association flag in JSON or YAML output which
was never a valid flag.
* sacctmgr - Catch and reject attempts to pass invalid QOS flags.
* Avoid returning an "INVALID" QOS flag in JSON or YAML output which was
never a valid flag.
* slurmdbd.conf can have permissions of 640 or 600.
* Avoid possible race condition that could cause a hang if process should
shutdown before all conmgr worker threads have started.
* Add HttpParserType parameter to slurm.conf
* Add UrlParserType parameter to slurm.conf
* Add http_parser/libhttp_parser plugin.
* slurmrestd - Switch to using http_parser/libhttp_parser plugin for http
processing.
* topology/block - Add new TopologyParam=BlockAsNodeRank option to reorder
nodes based on block layout. This can be useful if the naming convention
for the nodes does not natually map to the network topology.
* Fix memory leak in acct_gather_profile_influxdb.c.
* Allow --segment to be bigger than base block size.
* sacctmgr - Allow an operator to alter the allocated TRES of a job.
* sacctmgr - add new option to set fixed runaway jobs as FAILED or
COMPLETED.
* Job option --hint is now mutually exclusive with both --cores-per-socket
and --sockets-per-node.
* Add HealthCheckNodeState=START_ONLY option.
* Add a new CliFilterParameters=cli_filter_lua_path= to slurm.conf, enabling
the configuration of an absolute path to the cli_filter.lua script when
cli_filter/lua is configured.
* Allow removing a user's default association when AllowNoDefAcct=yes.
* scontrol - Fix "KillOnInvalidDependent" typo in show jobs output.
* scontrol - Use whitespace to separate all key-value pairs in show jobs
output. Previously, commas were used between job flags.
* scontrol - Fix GresAllowTaskSharing not always appearing in show jobs
output.
* slurmrestd - Log URLs that fail to parse under the debugflag=data instead
as errors or debug5 as invalid user provided is not directly an error of
the slurmrestd daemon itself.
* Fix use-cases incorrectly rejecting job requests when
MaxCPUsPer[Socket|Node] applied and CPUSpecList/CoreSpecCount configured.
* slurmrestd - Specify listening on UNIX socket by giving the "unix://"
prefix instead of "unix:" prefix for the URL scheme.
* Don't subtract 128 from sacct returned exit codes for the codes above 128.
* Fix issue resolving socket max segment size (MSS) via kernel to more
accurately size buffered communications.
* mpi/pmix - Restore the ability to follow symlinks on PMIx cli/lib temporal
directory creation when considered trusted (i.e. SlurmdSpoolDir, TmpFS,
PMIxCliTmpDirBase). This ability was lost in Slurm 23.02.6.
* Invalid use of -= or += aborts sacctmgr commands with an error.
* Add SLURM_JOB_SEGMENT_SIZE environment variable to salloc and srun job
environments when --segment is set.
* Remove the trailing space after AdminComment, SystemComment, Comment, and
Extra fields in scontrol show jobs output.
* slurmdbd - Make sure locks are appropriate when modifying or removing
associations while filtering by QOS.
* defaultqos is not set when loaded from dump file
* Add display for default qos when listing users and not specifying
"withassoc"
* slurmctld - Always close rejected incoming RPC connections to avoid
reading another incoming RPC request.
* slurmctld - Log runtime errors from global section of Lua scripts during
loading.
* Allow gpu shared gres to use RestrictedCoresPerGPU.
* slurmrestd - Fix memory leak that happened when submitting a request body
containing the "warnings", "errors", or "meta" field. This affects the
following endpoints: 'POST /slurmdb/v0.0.4*/qos'
* slurmrestd - Prevent triggering a fatal abort when parsing a non-empty
group id string. This affects all endpoints with request bodies containing
openapi_meta_client group field. It also affects the following endpoints:
+ 'GET /slurmdb/v0.0.4*/jobs'
+ 'POST /slurm/v0.0.4*/job/submit'
+ 'POST /slurm/v0.0.4*/job/{job_id}'
+ 'POST /slurm/v0.0.4*/job/allocate'
* Add the ability to gate access to a reservation by QOS.
* Add the ability to gate access to a reservation by Partition.
* scontrol - Add "SubmitLine" field to show job subcommand's output.
* slurmctld - Prevent an invalid read and a possible crash by rejecting any
arbitrary distribution jobs that do not specify a task count equal to the
number of node names in their node list. This does not affect srun,
salloc, or sbatch if -n is not used since they set the default task count.
* slurmrestd - Load slurmctld and slurmdbd OpenAPI plugins by default when
slurmdbd accounting is configured or only the slurmctld OpenAPI plugin
otherwise.
* slurmrestd - Improve error response message when URL is not found or not
supported.
* Fix race condition that could result in a syslog message being lost right
immediately before executing a script.
* Fix race condition that could result in log or scheduling log message
being lost or misdirected immediately before executing a script.
* Add support for "+=" and "-=" syntax when modifying TRES options via
"sacctmgr modify". Supported for QoS, User, Account and Cluster entities.
* slurmctld - Avoid falsely setting partially closed sockets as stale.
* slurmctld - Check if persistent connections have gone stale before
attempting to respond to RPC requests.
* slurmctld - Enforce stale connection check on RPCs with asynchronous I/O.
* TaskProlog and TaskEpilog no longer have to be fully qualified pathnames.
* When configless is enabled, TaskProlog and TaskEpilog scripts can now be
pushed to slurmds if the script files are in the same directory as
slurm.conf.
* When replacing nodes due to REPLACE flag, do not pull nodes from MAINT
reservations.
* Add a new --spread-segments option to salloc/sbatch/srun.
* tls/s2n - Fix PMIx server not working.
* slurmrestd - Add several more verbose errors for rejected requests in
slurmrestd's logs instead of "Unspecified error".
* slurmrestd - Add specific error when Content-Length header is too large
for any HTTP request.
* slurmrestd - Fix a bug in the "GET /slurm/v0.0.4*/node/{node_name}"
endpoint where the node's partitions field would be incorrectly populated.
* Log Lua backtrace on Lua script failures while loading scripts when
debugflags=script and at at least log level debug when Slurm is linked to
all Lua releases after version 5.4.6.
* Add 'MaxPurgeLimit' to 'slurmdbd.conf', replacing constant 50,000 limit.
* slurmrestd - Avoid logging redundant "Unspecified error" after failing to
load TLS certificates during startup.
* Fix gres enforce-binding not allowing multi-socket tasks with gres-per-
task requests.
* Add limited support for the conjunction of "--segment" and "--nodelist"
("-w") for topology/block.
* squeue - Add new --running-over and --running-under options, used to
filter jobs based on their runtime.
* Fix the 'is-busy' check for a node shared between partitions.
* Fix incorrect use of bind() on BSD platforms.
* Fix handling of strigger --fail/-F event.
* slurmctld - Check connections can read and write when testing if
connection has gone stale for incoming RPCs.
* Fix power save scripts that close stderr and stdout from being killed at
shutdown after 10 second timeout.
* slurmctld - Avoid edge cases causing file descriptors to never close when
an incoming RPC connection fails.
* slurmctld - Prevent the gang scheduler from hitting a segfault on restart,
reconfigure, or when using 'scontrol delete partition' if all partitions
are removed.
* data_parser/v0.0.44 - Added missing node state flag BLOCKED to parsable
node states. This affects 'scontrol show nodes --{json|yaml}', 'sinfo
- -{json|yaml}', and the following REST API endpoints:
+ 'GET /slurm/v0.0.44/nodes'
+ 'GET /slurm/v0.0.44/node/{node_name}'
* scontrol - Fix ping showing the wrong hostname when using the --cluster
option. The correct cluster name will now be displayed in the output when
using --cluster.
* Job stderr string will be blank in data_parser output, scontrol and sview
unless it has been explicitly set in the job. Users must assume that
stderr will default to the stdout path in that case.
* Use the node_name instead of hostname in file name creation.
* Set CgroupPlugin=disabled by default when built without cgroup support.
* CgroupPlugin=Autodetect will always return "disabled" when Slurm is built
without cgroup support.
* Do not build cgroup plugins on OpenBSD.
* mpi/pmix - The PMIx server will no longer store the node's HWLOC topology
in the job-level information under the PMIX_LOCAL_TOPO key by default. The
new PMIxShareServerTopology mpi.conf option can be used to share the
topology in PMIx v4+.
* slurmctld/slurmd - Fatal instead of crashing if no blocks or switches are
defined in topology.conf when using topology/tree or topology/block.
* slurmrestd - Honor HTTP "Connection: Keep-Alive" header to enable
persistent connections for HTTP/1.1 connections.
* slurmrestd - Reject HTTP Keep-Alive headers to enable persistent
connections for HTTP/0.9 connections.
* slurmrestd - Ignore HTTP "Connection: Keep-Alive" headers for HTTP/1.1
connections as they are persistent connections without "Connection: Close"
headers by default.
* slurmrestd - Ignore HTTP "Keep-Alive: *" headers as no valid parameters
were ever defined instead of logging a warning.
* slurmrestd - Replies to unparsable or unsupported requests will be
HTTP/1.1 formatted.
* Reimplement fs/lustre stats gathering to ensure it only accounts for usage
since the beginning of the job, as it was previously incorrectly reporting
usage since last stats reset. Also add the ability to gracefully handle
reset/overflow of the Lustre counters.
* Remove node_features/knl_generic plugin.
* Limit conmgr_threads to 256 for all configuration parameters.
* slurmctld - Avoid potentially logging invalid address for info level
logging of "slurmctld_req: received opcode %s from %pA uid %u".
* slurmrestd - Allow externally signed TLS certificates to be used for
https:// sockets. Certificate no longer needs to be signed by the
ca_cert_file that is used for internal Slurm TLS communications.
* Add a new 'reconfig_on_restart' to 'SlurmctldParameters' making a restart
of slurmctld (a process restart that was not triggered by "scontrol
reconfigure") trigger a reconfigure of all slurmd and sackd daemons.
* Skip attempting to resolve reserved user "nobody" (uid:99) using
getpwduid_r().
* Skip attempting to resolve reserved group "nobody" (uid:99) using
getgrgid_r().
* Always resolve group id 99 as group "nobody".
* Always resolve user id 99 as user "nobody".
* slurmctld,slurmd,slurmstepd,slurmdbd,slurmrestd,scrun: Avoid SEGFAULT
during shutdown due to race condition with receiving POSIX signal when
debugflags=conmgr.
* Avoid race condition when handling a TLS wrapped connection when the TLS
plugin is not loaded.
* switch/nvidia_imex - Fix channel count being one less than the actual
imex_channel_count configuration.
* switch/nvidia_imex - Add --network=unique-channel-per-segment option to
allocate a unique IMEX channel per segment inside a job when using the
topology/block plugin.
* switch/nvidia_imex - When the allocated number of IMEX channels hits
imex_channel_count, new jobs will start pending instead of running without
creating any IMEX channels.
* slurmrestd - Add environment variables SLURMRESTD_DEBUG_SYSLOG and
SLURMRESTD_DEBUG_STDERR to allow control of logging targets.
* slurmrestd - Add support for the SLURM_DEBUG environment variable.
* Add a new --consolidate-segments option to salloc/sbatch/srun.
* openapi/util - Add openapi/util plugin for request not requiring a
connection to the slurmctld or slurmdbd.
* slurmrestd - Load the openapi/util plugin by default.
* slurmctld,sackd,slurmdbd,slurmrestd: Limit the automatically detected
conmgr thread count to 32. Avoid having too many threads on higher core
count hosts.
* slurmd - Change default to SlurmdParameters=conmgr_threads=6.
* scrun - Change default conmgr thread count to 3.
* slurmstepd - Change default conmgr_threads=4 when
SlurmdParameters=conmgr_threads=# is not defined.
* slurmctld,slurmd,slurmstepd,sackd,slurmdbd,slurmrestd: Log warnings when
configured conmgr thread counts are outside of suggested range based on
detected CPU counts.
* slurmd - Add --parameters option to append additional options to the
SlurmdParameters option from slurm.conf. Can also be used with 'slurmd -C'
to print alternate cpu topologies. (E.g., 'slurmd -C --parameters
l3cache_as_socket'.)
* Add per-node Parameters option, allowing node-specific SlurmdParameters.
* Manually updating a node's state to IDLE with 'scontrol update nodename'
is no longer allowed if the node is in state REBOOT_ISSUED. This is to
prevent the REBBOOT_ISSUED state from not automatically clearing once the
reboot finishes.
* openapi/util - Add endpoints to convert hostlist expressions to an array
of hostnames and vice versa:
+ 'POST /util/v0.0.44/hostnames'
+ 'POST /util/v0.0.44/hostlist'
* slurmrestd - Return HTTP_STATUS_CODE_ERROR_UNPROCESSABLE_CONTENT if
querying an endpoint results in ESLURM_DATA_PATH_NOT_FOUND error.
* data_parser/v0.0.44 - Avoid hitting an xassert by return a DATA_TYPE_LIST
on error when dumping a HOSTLIST_STRING without complex mode instead of
returning DATA_TYPE_NULL.
* slurmdbd - Allow slurmdbd.conf to be owned by either SlurmUser or root
instead of just SlurmUser.
* Add new DisableArchiveCommands parameter to slurmdbd.conf to disable
"sacctmgr archive" commands.
* Prevent jobs submitted to multiple partitions from being killed when
removing a partition from slurm.conf and restarting or reconfiguring the
slurmctld. Pending jobs will remain pending with the remaining partitions.
If a job's running partition, or all its partitions, are removed, the job
will get killed with JobState=NODE_FAIL and Reason=PartitionDown. Jobs
killed due to removing partitions from slurm.conf cannot be requeued.
* slurmctld - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* slurmscriptd - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* slurmstepd - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* slurmrestd - Debug Log connection manager diagnostics on receiving POSIX
signal SIGPROF.
* Allow external jobs to be submitted without requiring the current working
directory be set.
* Fix srun ntasks calculation when nodes are requested using a min-max range
and --ntasks is not used.
* Add CgroupSlice option to cgroup.conf to allow changing the default
"system.slice" slice when starting slurmd and slurmstepd. If slurmd is
started with a unit file, modifying the unit and adding Slice= into the
[Service] section is needed. After making this change in cgroup.conf, the
old slurmstepd scopes need to be terminated before starting slurmd again.
* Reject invalid dependencies with more than one dependency separator type.
* Reject dependency specifications with trailing garbage as invalid syntax.
* Mark afterok/afternotok dependencies on a non-existent remote job id as
failed. Previously the dependency would remain unfulfilled indefinitely.
* Federated clusters update every registered remote dependency, including
duplicates. This prevents jobs from pending forever due to some
dependencies never being fulfilled.
* Fix aftercorr dependency on remote jobs in a federation.
* srun - Fix not accepting incoming TTY connection from slurmstepd due to
race condition with POSIX signals during step startup.
* Add MetricsType option to slurm.conf.
* Add slurmdb endpoints to update job database:
+ 'POST /slurmdb/v0.0.44/job/{jobid}'
+ 'POST /slurmdb/v0.0.44/jobs'
* Add CommunicationParameters=disable_http to slurm.conf.
* slurmctld - Load http_parser, url_parser and TLS plugins by default when
CommunicationParameters=disable_http is not set in slurm.conf.
* slurmctld,slurmd - Add the following HTTP endpoints:
+ GET /
+ GET /livez
+ GET /readyz
+ GET /healthz
* acct_gather_profile/influxdb - Add new acct_gather.conf option of
ProfileInfluxDBFrequency to configure the frequency at which data is sent
to InfluxDB (or disable buffering entirely if set to 0). Defaults to 30s.
* Fix slurmdbd error triggered by "sreport user topusage" when trying to get
data from monthly usage tables.
* Fix systemd potentially losing track of the daemon PIDs.
* Fix fs_factor being 0 in the rest API when using
PriorityFalgs=NO_FAIR_TREE.
* slurmctld - Enforce stale connection checks for rpc_queue enabled
messages.
* switch/nvidia_imex - IMEX channels are now accessible in the batch step
and interactive step.
* Validate QOS configuration even if AccountingStorageEnforce QOS is not
set.
* Validate AllowQOS, DenyQOS have valid QOS's listed.
* Change schedule_cycle_sum to uint64_t to prevent overflows which will
otherwise occur after 2^32 microseconds (71.6 minutes).
* scontrol - Fix regression where "scontrol update jobid= qos=" was not
considered a valid command.
* When using CR_CPU, prevent counting more cpus against a step then was
allocated.
* slurmd - Load http_parser, url_parser and TLS plugins by default when
CommunicationParameters=disable_http is not set in slurm.conf.
* sackd - Add --jwks-file and --key-file options to specify a non-default
absolute path to .jwks and .key authentication key(s) file.
* sinfo --state/-t flag now outputs a more helpful error message when passed
an invalid node state.
* When using CR_CPU, prevent counting more cpus against a step then was
allocated.
* Fix CPU over-allocation with enforce-binding when ntasks is specified.
* data_parser/v0.0.44 - Add nodes field to slurm_license_info_t parser.
* Add a new mode of operation to HRes (MODE_3). In this mode, resources are
allocated on a per-node basis, with the total consumption being calculated
from the most granular layer and summed up into the successively higher
layers.
* scontrol - return error when attempting to add node to a non-existent
topology block.
* Allow cloud nodes to dynamically set topology at registration with slurmd
- -conf="topology=...".
* Add HTTP endpoints to expose Slurm metrics:
+ GET /metrics/jobs
+ GET /metrics/nodes
+ GET /metrics/partitions
+ GET /metrics/scheduler
+ GET /metrics/user-accts
* switch/hpe_slingshot - Reduce sharing of RGIDs on multi-NIC connected
systems. Default to distributing RGIDs evenly across all available NICs.
* switch/hpe_slingshot - Add nic_distribution_count option to
SwitchParameters.
* switch/hpe_slingshot - Add nic_distribution_count parameter to the
- -network job and step option.
* slurmstepd - Prevent the slurmstepd from segfaulting if the
switch/hpe_slingshot plugin is enabled and SwitchParameters is not
specified.
* openapi/slurmctld - Fix 'DELETE /slurm/v0.0.XX/node/{node_name}' error
message so that it says an issue occurred while deleting instead of
updating.
* Fix segfault with multi-node gres with PrologFlags=RunInJob.
* accounting_storage/mysql - Improve performance for 'sacct -T/--truncate'
by skipping suspend table queries for unsuspended jobs.
* Change job_container/tmpfs plugin to be named namespace/tmpfs.
* Rename DebugFlags=JobContainer to DebugFlags=Namespace
* Add new namespace/linux plugin.
* slurmctld - Close connection earlier when reply on RPC enqueuing fails.
* Add SlurmctldParameters=enable_async_reply to slurm.conf.
* Add new "scancel --admin-comment" to allow admins to set the AdminComment
while canceling jobs.
* Add support for database password rotation in slurmdbd through new
StoragePassScript option.
* Add support for database password rotation for jobcomp/mysql through new
JobCompPassScript option.
* Add interactive_step_set_cpu_freq parameter to the list of
LaunchParameters in slurm.conf. This allows for the default cpu freq
governor to be set on interactive steps.
* Mon May 26 2025 Egbert Eich <eich@suse.com>
- Update to version 24.11.5
* Fix security issue where a coordinator could add a user with
elevated privileges (CVE-2025-43904, bsc#1243666).
* Return error to `scontrol` reboot on bad nodelists.
* `slurmrestd` - Report an error when QOS resolution fails for
v0.0.40 endpoints.
* `slurmrestd` - Report an error when QOS resolution fails for
v0.0.41 endpoints.
* `slurmrestd` - Report an error when QOS resolution fails for
v0.0.42 endpoints.
* `data_parser/v0.0.42` - Added `+inline_enums` flag which
modifies the output when generating OpenAPI specification.
It causes enum arrays to not be defined in their own schema
with references (`$ref`) to them. Instead they will be dumped
inline.
* Fix binding error with `tres-bind map/mask` on partial node
allocations.
* Fix `stepmgr` enabled steps being able to request features.
* Reject step creation if requested feature is not available
in job.
* `slurmd` - Restrict listening for new incoming RPC requests
further into startup.
* `slurmd` - Avoid `auth/slurm` related hangs of CLI commands
during startup and shutdown.
* `slurmctld` - Restrict processing new incoming RPC requests
further into startup. Stop processing requests sooner during
shutdown.
* `slurmcltd` - Avoid auth/slurm related hangs of CLI commands
during startup and shutdown.
* `slurmctld` - Avoid race condition during shutdown or
ereconfigure that could result in a crash due delayed
processing of a connection while plugins are unloaded.
* Fix small memleak when getting the job list from the database.
* Fix incorrect printing of `%` escape characters when printing
stdio fields for jobs.
* Fix padding parsing when printing stdio fields for jobs.
* Fix printing `%A` array job id when expanding patterns.
* Fix reservations causing jobs to be held for `Bad Constraints`.
* `switch/hpe_slingshot` - Prevent potential segfault on failed
curl request to the fabric manager.
* Fix printing incorrect array job id when expanding stdio file
names. The `%A` will now be substituted by the correct value.
* Fix printing incorrect array job id when expanding stdio file
names. The `%A` will now be substituted by the correct value.
* `switch/hpe_slingshot` - Fix VNI range not updating on slurmctld
restart or reconfigre.
* Fix steps not being created when using certain combinations of
`-c` and `-n` inferior to the jobs requested resources, when
using stepmgr and nodes are configured with
`CPUs == Sockets*CoresPerSocket`.
* Permit configuring the number of retry attempts to destroy CXI
service via the new destroy_retries `SwitchParameter`.
* Do not reset `memory.high` and `memory.swap.max` in slurmd
startup or reconfigure as we are never really touching this
in `slurmd`.
* Fix reconfigure failure of slurmd when it has been started
manually and the `CoreSpecLimits` have been removed from
`slurm.conf`.
* Set or reset CoreSpec limits when slurmd is reconfigured and
it was started with systemd.
* `switch/hpe-slingshot` - Make sure the slurmctld can free
step VNIs after the controller restarts or reconfigures while
the job is running.
* Fix backup `slurmctld` failure on 2nd takeover.
- Changes from version 24.11.4
* `slurmctld`,`slurmrestd` - Avoid possible race condition that
could have caused process to crash when listener socket was
closed while accepting a new connection.
* `slurmrestd` - Avoid race condition that could have resulted
in address logged for a UNIX socket to be incorrect.
* `slurmrestd` - Fix parameters in OpenAPI specification for the
following endpoints to have `job_id` field:
```
GET /slurm/v0.0.40/jobs/state/
GET /slurm/v0.0.41/jobs/state/
GET /slurm/v0.0.42/jobs/state/
GET /slurm/v0.0.43/jobs/state/
```
* `slurmd` - Fix tracking of thread counts that could cause
incoming connections to be ignored after burst of simultaneous
incoming connections that trigger delayed response logic.
* Avoid unnecessary `SRUN_TIMEOUT` forwarding to `stepmgr`.
* Fix jobs being scheduled on higher weighted powered down nodes.
* Fix how backfill scheduler filters nodes from the available
nodes based on exclusive user and `mcs_label` requirements.
* `acct_gather_energy/{gpu,ipmi}` - Fix potential energy
consumption adjustment calculation underflow.
* `acct_gather_energy/ipmi` - Fix regression introduced in 24.05.5
(which introduced the new way of preserving energy measurements
through slurmd restarts) when `EnergyIPMICalcAdjustment=yes`.
* Prevent `slurmctld` deadlock in the assoc mgr.
* Fix memory leak when `RestrictedCoresPerGPU` is enabled.
* Fix preemptor jobs not entering execution due to wrong
calculation of accounting policy limits.
* Fix certain job requests that were incorrectly denied with
node configuration unavailable error.
* `slurmd` - Avoid crash due when slurmd has a communications
failure with `slurmstepd`.
* Fix memory leak when parsing yaml input.
* Prevent `slurmctld` from showing error message about `PreemptMode=GANG`
being a cluster-wide option for `scontrol update part` calls
that don't attempt to modify partition PreemptMode.
* Fix setting `GANG` preemption on partition when updating
`PreemptMode` with `scontrol`.
* Fix `CoreSpec` and `MemSpec` limits not being removed
from previously configured slurmd.
* Avoid race condition that could lead to a deadlock when `slurmd`,
`slurmstepd`, `slurmctld`, `slurmrestd` or `sackd` have a fatal
event.
* Fix jobs using `--ntasks-per-node` and `--mem` keep pending
forever when the requested mem divided by the number of CPUs
will surpass the configured `MaxMemPerCPU`.
* `slurmd` - Fix address logged upon new incoming RPC connection
from `INVALID` to IP address.
* Fix memory leak when retrieving reservations. This affects
`scontrol`, `sinfo`, `sview`, and the following `slurmrestd`
endpoints:
`GET /slurm/{any_data_parser}/reservation/{reservation_name}`
`GET /slurm/{any_data_parser}/reservations`
* Log warning instead of `debuflags=conmgr` gated log when
deferring new incoming connections when number of active
connections exceed `conmgr_max_connections`.
* Avoid race condition that could result in worker thread pool
not activating all threads at once after a reconfigure resulting
in lower utilization of available CPU threads until enough
internal activity wakes up all threads in the worker pool.
* Avoid theoretical race condition that could result in new
incoming RPC
socket connections being ignored after reconfigure.
* slurmd - Avoid race condition that could result in a state
where new incoming RPC connections will always be ignored.
* Add ReconfigFlags=KeepNodeStateFuture to restore saved `FUTURE`
node state on restart and reconfig instead of reverting to
`FUTURE` state. This will be made the default in 25.05.
* Fix case where hetjob submit would cause `slurmctld` to crash.
* Fix jobs using `--cpus-per-gpu` and `--mem` keep pending forever
when the requested mem divided by the number of CPUs will surpass
the configured `MaxMemPerCPU`.
* Enforce that jobs using `--mem` and several `--*-per-*` options
do not violate the `MaxMemPerCPU` in place.
* `slurmctld` - Fix use-cases of jobs incorrectly pending held
when `--prefer` features are not initially satisfied.
* `slurmctld` - Fix jobs incorrectly held when `--prefer` not
satisfied in some use-cases.
* Ensure `RestrictedCoresPerGPU` and `CoreSpecCount` don't overlap.
- Fix backward compatibility fallout from last update.
* Thu Apr 24 2025 Christian Goll <cgoll@suse.com>
- removed openmpi4-hpc dependency for test suite.
* Fri Mar 07 2025 Atri Bhattacharya <badshah400@gmail.com>
- Update to version 24.11.3:
* Fix database cluster ID generation not being random.
* Fix a regression in which `slurmd -G` gave no output.
* Fix a long-standing crash in `slurmctld` after updating a
reservation with an empty nodelist. The crash could occur
after restarting slurmctld, or if downing/draining a node
in the reservation with the `REPLACE` or `REPLACE_DOWN` flag.
* Avoid changing process name to "`watch`" from original daemon name.
This could potentially breaking some monitoring scripts.
* Avoid `slurmctld` being killed by `SIGALRM` due to race condition
at startup.
* Fix race condition in slurmrestd that resulted in "`Requested
data_parser plugin does not support OpenAPI plugin`" error being
returned for valid endpoints.
* Fix race between `task/cgroup` CPUset and `jobacctgather/cgroup`.
The first was removing the pid from `task_X` cgroup directory
causing memory limits to not being applied.
* If multiple partitions are requested, set the `SLURM_JOB_PARTITION`
output environment variable to the partition in which the job is
running for `salloc` and `srun` in order to match the documentation
and the behavior of `sbatch`.
* `srun` - Fixed wrongly constructed `SLURM_CPU_BIND` env variable
that could get propagated to downward srun calls in certain mpi
environments, causing launch failures.
* Don't print misleading errors for stepmgr enabled steps.
* `slurmrestd` - Avoid connection to slurmdbd for the following
endpoints:
```
GET /slurm/v0.0.41/jobs
GET /slurm/v0.0.41/job/{job_id}
```
* `slurmrestd` - Avoid connection to slurmdbd for the following
endpoints:
```
GET /slurm/v0.0.40/jobs
GET /slurm/v0.0.40/job/{job_id}
```
* `slurmrestd` - Fix possible memory leak when parsing arrays with
`data_parser/v0.0.40`.
* `slurmrestd` - Fix possible memory leak when parsing arrays with
`data_parser/v0.0.41`.
* `slurmrestd` - Fix possible memory leak when parsing arrays with
`data_parser/v0.0.42`.
- Changes from version 24.11.2:
* Fix segfault when submitting `--test-only` jobs that can
preempt.
* Fix regression introduced in 23.11 that prevented the
following flags from being added to a reservation on an
update: `DAILY`, `HOURLY`, `WEEKLY`, `WEEKDAY`, and `WEEKEND`.
* Fix crash and issues evaluating job's suitability for running
in nodes with already suspended job(s) there.
* `slurmctld` will ensure that healthy nodes are not reported as
`UnavailableNodes` in job reason codes.
* Fix handling of jobs submitted to a current reservation with
flags `OVERLAP,FLEX` or `OVERLAP,ANY_NODES` when it overlaps nodes
with a future maintenance reservation. When a job submission
had a time limit that overlapped with the future maintenance
reservation, it was rejected. Now the job is accepted but
stays pending with the reason "`ReqNodeNotAvail, Reserved for
maintenance`".
* `pam_slurm_adopt` - avoid errors when explicitly setting some
arguments to the default value.
* Fix QOS preemption with `PreemptMode=SUSPEND`.
* `slurmdbd` - When changing a user's name update lineage at the
same time.
* Fix regression in 24.11 in which `burst_buffer.lua` does not
inherit the `SLURM_CONF` environment variable from `slurmctld` and
fails to run if slurm.conf is in a non-standard location.
* Fix memory leak in slurmctld if `select/linear` and the
`PreemptParameters=reclaim_licenses` options are both set in
`slurm.conf`. Regression in 24.11.1.
* Fix running jobs, that requested multiple partitions, from
potentially being set to the wrong partition on restart.
* `switch/hpe_slingshot` - Fix compatibility with newer cxi
drivers, specifically when specifying `disable_rdzv_get`.
* Add `ABORT_ON_FATAL` environment variable to capture a backtrace
from any `fatal()` message.
* Fix printing invalid address in rate limiting log statement.
* `sched/backfill` - Fix node state `PLANNED` not being cleared from
fully allocated nodes during a backfill cycle.
* `select/cons_tres` - Fix future planning of jobs with
`bf_licenses`.
* Prevent redundant "`on_data returned rc: Rate limit exceeded,
please retry momentarily`" error message from being printed in
slurmctld logs.
* Fix loading non-default QOS on pending jobs from pre-24.11
state.
* Fix pending jobs displaying `QOS=(null)` when not explicitly
requesting a QOS.
* Fix segfault issue from job record with no `job_resrcs`.
* Fix failing `sacctmgr delete/modify/show` account operations
with `where` clauses.
* Fix regression in 24.11 in which Slurm daemons started
catching several `SIGTSTP`, `SIGTTIN` and `SIGUSR1` signals and
ignored them, while before they were not ignoring them. This
also caused slurmctld to not being able to shutdown after a
`SIGTSTP` because slurmscriptd caught the signal and stopped
while slurmctld ignored it. Unify and fix these situations and
get back to the previous behavior for these signals.
* Document that `SIGQUIT` is no longer ignored by `slurmctld`,
`slurmdbd`, and slurmd in 24.11. As of 24.11.0rc1, `SIGQUIT` is
identical to `SIGINT` and `SIGTERM` for these daemons, but this
change was not documented.
* Fix not considering nodes marked for reboot without ASAP in
the scheduler.
* Remove the `boot^` state on unexpected node reboot after return
to service.
* Do not allow new jobs to start on a node which is being
rebooted with the flag `nextstate=resume`.
* Prevent lower priority job running after cancelling an ASAP
reboot.
* Fix srun jobs starting on `nextstate=resume` rebooting nodes.
- Sync upgrades file to reflect last updated versions.
- Pass `-DH5_USE_112_API -DDH5Oget_info_vers=1` to CFLAGS to allow
building with hdf5 1.14 as slurm does not yet support HDF5 v114
API.
* Fri Feb 07 2025 Egbert Eich <eich@suse.com>
- Update to version 24.11.1:
* With client commands `MIN_MEMORY` will show `mem_per_tres` if
specified.
* Fix errno message about bad constraint.
* `slurmctld` - Fix crash and possible split brain issue if the
backup controller handles an scontrol reconfigure while in control
before the primary resumes operation.
* Fix `stepmgr` not getting dynamic node addrs from the controller
* `stepmgr` - avoid `Unexpected missing socket` errors.
* Fix `scontrol show steps` with dynamic stepmgr.
* Deny jobs using the `R:` option of `--signal` if `PreemptMode=OFF`
globally.
* Force jobs using the `R:` option of `--signal` to be
preemptable.
by requeue or cancel only. If `PreemptMode` on the partition or
QOS is off or suspend, the job will default to using
`PreemptMode=cancel`.
* If `--mem-per-cpu` exceeds `MaxMemPerCPU`, the number of CPUs
per task will always be increased even if --cpus-per-task was
specified. This is needed to ensure each task gets the expected
amount of memory.
* Fix compilation issue on OpenSUSE Leap 15.
* Fix jobs using more nodes than needed when not using `-N`.
* Fix issue with allocation being allocated less resources.
than needed when using `--gres-flags=enforce-binding`.
* `select/cons_tres` - Fix errors with `MaxCpusPerSocket`
partition limit. Used CPUs/cores weren't counted properly,
nor limiting free ones to avail, when the socket was partially
allocated, or the job request went beyond this limit.
* Fix issue when jobs were preempted for licenses even if there
were enough licenses available.
* Fix `srun` `ntasks` calculation inside an allocation when nodes
are requested using a min-max range.
* Print correct number of digits for `TmpDisk` in `sdiag`.
* Fix a regression in 24.11 which caused file transfers to a job
with sbcast to not join the job container namespace.
* `data_parser/v0.0.40` - Prevent a segfault in the `slurmrestd`
when dumping data with v0.0.40+complex data parser.
* Remove logic to force lowercase GRES names.
* `data_parser/v0.0.42` - Prevent the association id from always
being dumped as NULL when parsing in complex mode. Instead it
will now dump the id. This affects the following endpoints:
- `GET slurmdb/v0.0.42/association`
- `GET slurmdb/v0.0.42/associations`
- `GET slurmdb/v0.0.42/config`
* Fixed a job requeuing issue that merged job entries into the
same SLUID when all nodes in a job failed simultaneously.
* When a job completes, try to give idle nodes to reservations with
the `REPLACE` flag before allowing them to be allocated to jobs.
* Avoid expensive lookup of all associations when dumping or
parsing for v0.0.42 endpoints.
* Avoid expensive lookup of all associations when dumping or
parsing for v0.0.41 endpoints.
* Avoid expensive lookup of all associations when dumping or
parsing for v0.0.40 endpoints.
* Fix segfault when testing jobs against nodes with invalid gres.
* Fix performance regression while packing larger RPCs.
* `job_container/tmpfs` - Fix Xauthoirty file being created.
outside the container when `EntireStepInNS` is enabled.
* `job_container/tmpfs` - Fix `spank_task_post_fork` not always
running in the container when `EntireStepInNS` is enabled.
* Fix a job potentially getting stuck in CG on permissions
errors while setting up X11 forwarding.
* Fix error on X11 shutdown if Xauthority file was not created.
* `slurmctld` - Fix memory or fd leak if an RPC is recieved that
is not registered for processing.
* Inject `OMPI_MCA_orte_precondition_transports` when using PMIx.
This fixes mpi apps using Intel OPA, PSM2 and OMPI 5.x when ran
through `srun`.
* Don't skip the first `partition_job_depth` jobs per partition.
* Fix gres allocation issue after controller restart.
* Fix issue where jobs requesting CPUs-per-GPU hang in queue.
* `switch/hpe_slingshot` - Treat HTTP status forbidden the same as
unauthorized, allowing for a graceful retry attempt.
- Slurmdbd no longer starts as root so that the systemd
`RuntimeDirectory` option works properly. This requires further
adjustments:
* Use /run/slurmdbd as directory for PID file. This is created
when slurmdbd starts by `RuntimeDirectory=slurmdbd` in the
systemd service file (boo#1236928).
* Upon installation, create empty `/var/log/slurmdbd.log` owned
by user slurm as slurmdbd is not able to create it (boo#1236929).
* Fri Jan 17 2025 Egbert Eich <eich@suse.com>
- Make test suite package work on SLE-12.
* Thu Jan 09 2025 Egbert Eich <eich@suse.com>
- Fix testsuite:
Cater for erroneous: `#include </src/[slurm_internal_header]>`
statements.
* Mon Jan 06 2025 Egbert Eich <eich@suse.com>
- Update to version 24.11
* `slurmctld` - Reject arbitrary distribution jobs that do not
specifying a task count.
* Fix backwards compatibility of the `RESPONSE_JOB_INFO RPC`
(used by `squeue`, `scontrol show job`, etc.) with Slurm clients
version 24.05 and below. This was a regression in 24.11.0rc1.
* Do not let `slurmctld`/`slurmd` start if there are more nodes
defined in `slurm.conf` than the maximum supported amount
(64k nodes).
* `slurmctld` - Set job's exit code to 1 when a job fails with
state `JOB_NODE_FAIL`. This fixes `sbatch --wait` not being able
to exit with error code when a job fails for this reason in
some cases.
* Fix certain reservation updates requested from 23.02 clients.
* `slurmrestd` - Fix populating non-required object fields of
objects as `{}` in JSON/YAML instead of `null` causing compiled
OpenAPI clients to reject the response to
`GET /slurm/v0.0.40/jobs` due to validation failure of
`.jobs[].job_resources`.
* Fix issue where older versions of Slurm talking to a 24.11 dbd
could loose step accounting.
* Fix minor memory leaks.
* Fix bad memory reference when `xstrchr` fails to find char.
* Remove duplicate checks for a data structure.
* Fix race condition in `stepmgr` step completion handling.
* `slurm.spec` - add ability to specify patches to apply on the
command line.
* `slurm.spec` - add ability to supply extra version information.
* Fix 24.11 HA issues.
* Fix requeued jobs keeping their priority until the decay thread
happens.
* Fix potential memory corruption in `select/cons_tres` plugin.
* Avoid cache coherency issue on non-x86 platforms that could
result in a POSIX signal being ignored or an abort().
* `slurmctld` - Remove assertion in development builds that would
trigger if an outdated client attempted to connect.
* `slurmd` - Wait for `PrologEpilogTimeout` on reconfigure for
prologs to finish. This avoids a situation where the slurmd
never detects that the prolog completed.
* `job_container/tmpfs` - Setup x11 forwarding within the namespace.
* `slurmctld` - fix memory leak when sending a `DBD_JOB_START`
message.
* Fix issue with accounting rollup dealing with association tables.
* Fix minor memory leaks.
* Fix potential thread safety issues.
* Init mutex in burst_buffer plugins.
* `slurmdbd` - don't log errors when no changes occur from db
requests.
* `slurmcltd`,`slurmd` - Avoid deadlock during reconfigure if too
many POSIX signals are received.
* Improve error type logged from partial or incomplete reading
from socket or pipe to avoid potentially logging an error from
a previous syscall.
* `slurmrestd` - Improve the handling of queries when unable to
connect to slurmdbd by providing responses when possible.
* `slurmrestd`,`sackd`,`scrun` - Avoid rare hangs related to I/O.
* `scrun` - Add support `--all` argument for kill subcommand.
* Remove `srun --cpu-bind=rank`.
* Add `resource_spec/cpus` and `resource_spec/memory` entry
points in data_parser to print the `CpuSpecList` and
`MemSpecLimit` in `sinfo --json`.
* `sinfo` - Add `.sinfo[].resource_spec.cpus` and
`.sinfo[].resource_spec.memory` fields to print the `CpuSpecList`
and `MemSpecLimit` dumped by `sinfo --{json|yaml}`.
* Increase efficency of sending logs to syslog.
* Switch to new official YAML mime type `application/yaml` in
compliance with RFC9512 as primary mime type for YAML formatting.
* `slurmrestd` - Remove deprecated fields from the following
endpoints:
`.result` from `POST /slurm/v0.0.42/job/submit`.
`.job_id`, `.step_id`, `.job_submit_user_msg` from `POST /slurm/v0.0.42/job/{job_id}`.
`.job.exclusive`, `.jobs[].exclusive` to `POST /slurm/v0.0.42/job/submit`.
`.jobs[].exclusive` from `GET /slurm/v0.0.42/job/{job_id}`.
`.jobs[].exclusive` from `GET /slurm/v0.0.42/jobs`.
`.job.oversubscribe`, `.jobs[].oversubscribe` to `POST /slurm/v0.0.42/job/submit`.
`.jobs[].oversubscribe` from `GET /slurm/v0.0.42/job/{job_id}`.
`.jobs[].oversubscribe` from `GET /slurm/v0.0.42/jobs`.
* `scontrol` - Removed deprecated fields `.jobs[].exclusive` and
`.jobs[].oversubscribe` from `scontrol show jobs --{json|yaml}`.
* `squeue` - Removed deprecated fields `.jobs[].exclusive` and
`.jobs[].oversubscribe` from `squeue --{json|yaml}`.
* Improve the way to run external commands and fork processes to
avoid non-async-signal safe calls between a fork and an exec.
We fork ourselves now and executes the commands in a safe
environment. This includes spank prolog/epilog executions.
* Improve `MaxMemPerCPU` enforcement when exclusive jobs request
per node memory and the partition has heterogeneous nodes.
* Remove a `TOCTOU` where multiple steps requesting an energy
reading at the same time could cause too frequent accesses
to the drivers.
* Limit `SwitchName` to `HOST_NAME_MAX` chars length.
* For `scancel --ctld` and the following rest api endpoints:
`DELETE /slurm/v0.0.40/jobs`
`DELETE /slurm/v0.0.41/jobs`
`DELETE /slurm/v0.0.42/jobs`
Support array expressions in the responses to the client.
* `salloc` - Always output node names to the user when an
allocation is granted.
* `slurmrestd` - Removed all v0.0.39 endpoints.
* `select/linear` - Reject jobs asking for GRES per
`job|socket|task` or `cpus|mem` per GRES.
* Add `/nodes` POST endpoint to REST API, supports multiple
node update whereas previously only single nodes could be
updated through `/node/<nodename>` endpoint:
`POST /slurm/v0.0.42/nodes`
* Do not allow changing or setting `PreemptMode=GANG` to a
partition as this is a cluster-wide option.
* Add `%b` as a file name pattern for the array task id modulo 10.
* Skip packing empty nodes when they are hidden during
`REQUEST_NODE_INFO RPC`.
* `accounting_storage/mysql` - Avoid a fatal condition when
the db server is not reachable.
* Always lay out steps cyclically on nodes in an allocation.
* `squeue` - add priority by partition
(`.jobs[].priority_by_partition`) to JSON and YAML output.
* `slurmrestd` - Add clarification to `failed to open slurmdbd
connection` error if the error was the result of an
authentication failure.
* Make it so `slurmctld` responds to RPCs that have authentication
errors with the `SLURM_PROTOCOL_AUTHENTICATION_ERROR` error
code.
* `openapi/slurmctld` - Display the correct error code instead
of `Unspecified error` if querying the following endpoints
fails:
`GET /slurm/v0.0.40/diag/`
`GET /slurm/v0.0.41/diag/`
`GET /slurm/v0.0.42/diag/`
`GET /slurm/v0.0.40/licenses/`
`GET /slurm/v0.0.41/licenses/`
`GET /slurm/v0.0.42/licenses/`
`GET /slurm/v0.0.40/reconfigure`
`GET /slurm/v0.0.41/reconfigure`
`GET /slurm/v0.0.42/reconfigure`
* Fix how used CPUs are tracked in a job allocation to allow the
max number of concurrent steps to run at a time if threads per
core is greater than 1.
* In existing allocations SLURM_GPUS_PER_NODE environment
variable will be ignored by srun if `--gpus` is specified.
* When using `--get-user-env` explicitly or implicitly, check
if PID or mnt namespaces are disabled and fall back to old
logic that does not rely on them when they are not available.
* Removed non-functional option `SLURM_PROLOG_CPU_MASK` from
`TaskProlog` which was used to reset the affinity of a task
based on the mask given.
* `slurmrestd` - Support passing of `-d latest` to load latest
version of `data_parser` plugin.
* `sacct`,`sacctmgr`,`scontrol`,`sdiag`,`sinfo`,`squeue`,`sshare`
- Change response to `--json=list` or `--yaml=list` to send
list of plugins to stdout and descriptive header to stderr to
allow for easier parsing.
* `slurmrestd` - Change response to `-d list`, `-a list` or
`-s list` to send list of plugins to stdout and descriptive
header to stderr to allow for easier parsing.
* `sacct`,`sacctmgr`,`scontrol`,`sdiag`,`sinfo`,`squeue`,
`sshare`,`slurmrestd` - Avoid crash when loading `data_parser`
plugins fail due to NULL dereference.
* Add autodetected GPUs to the output of `slurmd -C`
* Remove `burst_buffer/lua` call `slurm.job_info_to_string()`.
* Add `SchedulerParameters=bf_allow_magnetic_slot` option. It
allows jobs in magnetic reservations to be planned by backfill
scheduler.
* `slurmrestd` - Refuse to run as root, `SlurmUser`, and
`nobody(99)`.
* `openapi/slurmctld` - Revert regression that caused signaling
jobs to cancel entire job arrays instead of job array tasks:
`DELETE /slurm/v0.0.40/{job_id}`
`DELETE /slurm/v0.0.41/{job_id}`
`DELETE /slurm/v0.0.42/{job_id}`
* `openapi/slurmctld` - Support more formats for `{job_id}`
including job steps:
`DELETE /slurm/v0.0.40/{job_id}`
`DELETE /slurm/v0.0.41/{job_id}`
`DELETE /slurm/v0.0.42/{job_id}`
* Alter scheduling of jobs at submission time to consider job
submission time and job id. This makes it so that that
interactive jobs aren't allocated resources before batch jobs
when they have the same priority at submit time.
* Fix multi-cluster submissions with differing Switch plugins.
* `slurmrestd` - Change `+prefer_refs` flag to default in
`data_parser/v0.0.42` plugin. Add `+minimize_refs` flag to
inline single referenced schemas in the OpenAPI schema. This
sets the default OpenAPI schema generation behavior of
`data_parser/v0.0.42` to match v0.0.41 `+prefer_refs` and
v0.0.40 (without flags).
* Fix `LaunchParameters=batch_step_set_cpu_freq`.
* Clearer `seff` warning message for running jobs.
* `data_parser/v0.0.42` - Rename `JOB_INFO` field
`minimum_switches` to `required_switches` to reflect the
actual behavior.
* `data_parser/v0.0.42` - Rename `ACCOUNT_CONDITION` field
`assocation` to `association` to fix typo.
* `cgroup/v2` - fix cgroup cleanup when running inside a
container without write permissions to `/sys/fs/cgroup`.
* `cgroup/v2` - fix accounting of swap events detection.
* Fix gathering MaxRSS for jobs that run shorter than two
`jobacctgather` intervals. Get the metrics from cgroups
`memory.peak` or `memory.max_usage_in_bytes` where available.
* `openapi/slurmctld` - Set complex number support for the
following fields:
`.shares[][].fairshare.factor`
`.shares[][].fairshare.level`
for endpoints:
`GET /slurm/v0.0.42/shares`
and for commands:
`sshare --json`
`sshare --yaml`
* `data_parser/v0.0.42` - Avoid dumping `Infinity` for `NO_VAL`
tagged `number` fields.
* Add `TopologyParam=TopoMaxSizeUnroll=#` to allow
`--nodes=<min>-<max>` for `topology/block`.
* `sacct` - Respect `--noheader` for `--batch-script` and
`--env-vars`.
* `sacct` - Remove extra newline in output from `--batch-script`
and --env-vars.
* Add `sacctmgr ping` command to query status of `slurmdbd`.
* Generate an error message when a `NodeSet` name conflicts with
a `NodeName`, and prevent the controller from starting if such
a conflict exists.
* `slurmd` - properly detect slurmd restarts in the energy
gathering logic which caused bad numbers in accounting.
* `sackd` - retry fetching slurm configs indefinately in
configless mode.
* `job_submit/lua` - Add `assoc_qos` attribute to `job_desc`
to display all potential QOS's for a job's association.
* `job_submit/lua` - Add `slurm.get_qos_priority()` function
to retrieve the given QOS's priority.
* `sbcast` - Add `--nodelist` option to specify where files are
transmitted to.
* `sbcast` - Add `--no-allocation` option to transmit files to
nodes outside of a job allocation
* Add `DataParserParameters` `slurm.conf` parameter to allow
setting default value for CLI `--json` and `--yaml` arguments.
* `seff` - improve step's max memory consumption report by using
`TresUsageInTot` and `TresUsageInAve` instead of overestimating
the values.
* Enable RPC queueing for `REQUEST_KILL_JOBS`, which is used when
`scancel` is executed with `--ctld` flag.
* `slurmdbd` - Add `-u` option. This is used to determine if
restarting the DBD will result in database conversion.
* Fix `srun` inside an `salloc` in a federated cluster when using
IPv6.
* Calculate the forwarding timeouts according to tree depth
rather than node count / tree width for each level. Fixes race
conditions with same timeouts between two consecutive node
levels.
* Add ability to submit jobs with multiple QOS.
* Fix difference in behavior when swapping partition order in job
submission.
* Improve `PLANNED` state detection for mixed nodes and updating
state before yielding backfill locks.
* Always consider partition priority tiers when deciding to try
scheduling jobs on submit.
* Prevent starting jobs without reservations on submit when there
are pending jobs with reservations that have flags `FLEX` or
`ANY_NODES` that can be scheduled on overlapping nodes.
* Prevent jobs that request both high and low priority tier
partitions from starting on submit in lower priority tier
partitions if it could delay pending jobs in higher priority
tier partitions.
* `scontrol` - Wait for `slurmctld` to start reconfigure in
foreground mode before returning.
* Improve reconfigure handling on Linux to only close open file
descriptors to avoid long delays on systems with large
`RLIMIT_NOFILE` settings.
* `salloc` - Removed `--get-user-env` option.
* Removed the instant on feature from `switch/hpe_slingshot`.
* Hardware collectives in `switch/hpe_slingshot` now requires
`enable_stepmgr`.
* Allow backfill to plan jobs on nodes currently being used by
exclusive user or mcs jobs.
* Avoid miscaching IPv6 address to hostname lookups that could
have caused logs to have the incorrect hostname.
* `scontrol` - Add `--json`/`--yaml` support to `listpids`
* `scontrol` - Add `liststeps`
* `scontrol` - Add `listjobs`
* `slurmrestd` - Avoid connection to slurmdbd for the following
endpoints:
`GET /slurm/v0.0.42/jobs`
`GET /slurm/v0.0.42/job/{job_id}`
* `slurmctld` - Changed incoming RPC handling to dedicated thread
pool.
* `job_container/tmpfs` - Add `EntireStepInNS` option that will
place the `slurmstepd` process within the constructed namespace
directly.
* `scontrol show topo` - Show aggregated block sizes when using
`topology/block`.
* `slurmrestd` - Add more descriptive HTTP status for
authentication failure and connectivity errors with controller.
* `slurmrestd` - Improve reporting errors from `slurmctld` for
job queries:
`GET /slurm/v0.0.41/{job_id}`
`GET /slurm/v0.0.41/jobs/`
* Avoid rejecting a step request that needs fewer GRES than nodes
in the job allocation.
* `slurmrestd` - Tag the never populated `.jobs[].pid` field as
deprecated for the following endpoints:
`GET /slurm/v0.0.42/{job_id}`
`GET /slurm/v0.0.42/jobs/`
* `scontrol`,`squeue` - Tag the never populated `.jobs[].pid` field
as deprecated for the following:
`scontrol show jobs --json`
`scontrol show jobs --yaml`
`scontrol show job ${JOB_ID} --json`
`scontrol show job ${JOB_ID} --yaml`
`squeue --json`
`squeue --yaml`
* `data_parser` v0.0.42 - fix timestamp parsing regression
introduced in in v0.0.40 (eaf3b6631f), parsing of non iso 8601
style timestamps
* `cgroup/v2` will detect some special container and namespaced
setups and will work with it.
* Support IPv6 in configless mode.
* Add `SlurmctldParamters=ignore_constraint_validation` to ignore
`constraint/feature` validation at submission.
* `slurmrestd` - Set `.pings[].mode` field as deprecated in the
following endpoints:
`GET /slurm/v0.0.42/ping`
* `scontrol` - Set `.pings[].mode` field as deprecated in the
following commands:
`scontrol ping --json`
`scontrol ping --yaml`
* `slurmrestd` - Set `.pings[].pinged` field as deprecated in
the following endpoints:
`GET /slurm/v0.0.42/ping`
* `scontrol` - Set `.pings[].pinged` field as deprecated in the
following commands:
`scontrol ping --json`
`scontrol ping --yaml`
* `slurmrestd` - Add `.pings[].primary` field to the following
endpoints:
`GET /slurm/v0.0.42/ping`
* `scontrol` - Add `.pings[].primary` field to the following
commands:
`scontrol ping --json`
`scontrol ping --yaml`
* `slurmrestd` - Add `.pings[].responding` field to the following
endpoints:
`GET /slurm/v0.0.42/ping`
* `scontrol` - Add `.pings[].responding` field to the following
commands:
`scontrol ping --json`
`scontrol ping --yaml`
* Prevent jobs without reservations from delaying jobs in
reservations with flags `FLEX` or `ANY_NODES` in the main
scheduler.
* Fix allowing to ask for multiple different types of TRES
when one of them has a value of 0.
* `slurmctld` - Add a grace period to ensure the agent retry
queue is properly flushed during shutdown.
* Don't ship `src/slurmrestd/plugins/openapi/slurmdbd/openapi.json`
`slurmrest` should always be used to enerate a new OpenAPI
schema (aka openapi.json or openapi.yaml).
* `mpi/pmix` - Fix potential deadlock and races with het jobs,
and fix potential memory and FDs leaks.
* Fix jobs with `--gpus` being rejected in some edge cases for
partitions where not all nodes have the same amount of GPUs
and CPUs configured.
* In an extra constraints expression in a job request, do not
allow an empty string for a key or value.
* In an extra constraints expression in a job request, fix
validation that requests are separated by boolean operators.
* Add `TaskPluginParam=OOMKillStep` to kill the step as a whole
when one task OOMs.
* Fix `scontrol` show conf not showing all `TaskPluginParam`
elements.
* `slurmrestd` - Add fields `.job.oom_kill_step`
`.jobs[].oom_kill_step` to `POST /slurm/v0.0.42/job/submit`
and `POST /slurm/v0.0.42/job/allocate`.
* Improve performance for `_will_run_test()`.
* Add `SchedulerParameters=bf_topopt_enable` option to enable
experimental hook to control backfill.
* If a step fails to launch under certain conditions, set the
step's state to `NODE_FAIL`.
* `sched/backfill` - Fix certain situations where a job would
not get a planned time, which could lead to it being delayed
by lower priority jobs.
* `slurmrestd` - Dump JSON `null` instead of `{}` (empty object)
for non-required fields in objects to avoid client
compatiblity issues for v0.0.42 version tagged endpoints.
* `sacct`,`sacctmgr`,`scontrol`,`sdiag`,`sinfo`,`squeue`,
`sshare` - Dump `null` instead `{}` (empty object) for
non-required fields in objects to avoid client compatiblity
issues when run with `--json` or `--yaml`.
* Fri Nov 01 2024 Egbert Eich <eich@suse.com>
- Update to version 24.05.4 & fix for CVE-2024-48936.
* Fix generic int sort functions.
* Fix user look up using possible unrealized uid in the dbd.
* `slurmrestd` - Fix regressions that allowed `slurmrestd` to
be run as SlurmUser when `SlurmUser` was not root.
* mpi/pmix fix race conditions with het jobs at step start/end
which could make srun to hang.
* Fix not showing some `SelectTypeParameters` in `scontrol show
config`.
* Avoid assert when dumping removed certain fields in JSON/YAML.
* Improve how shards are scheduled with affinity in mind.
* Fix `MaxJobsAccruePU` not being respected when `MaxJobsAccruePA`
is set in the same QOS.
* Prevent backfill from planning jobs that use overlapping
resources for the same time slot if the job's time limit is
less than `bf_resolution`.
* Fix memory leak when requesting typed gres and
`--[cpus|mem]-per-gpu`.
* Prevent backfill from breaking out due to "system state
changed" every 30 seconds if reservations use `REPLACE` or
`REPLACE_DOWN` flags.
* `slurmrestd` - Make sure that scheduler_unset parameter defaults
to true even when the following flags are also set:
`show_duplicates`, `skip_steps`, `disable_truncate_usage_time`,
`run_away_jobs`, `whole_hetjob`, `disable_whole_hetjob`,
`disable_wait_for_result`, `usage_time_as_submit_time`,
`show_batch_script`, and or `show_job_environment`. Additionaly,
always make sure show_duplicates and
`disable_truncate_usage_time` default to true when the following
flags are also set: `scheduler_unset`, `scheduled_on_submit`,
`scheduled_by_main`, `scheduled_by_backfill`, and or `job_started`.
This effects the following endpoints:
`GET /slurmdb/v0.0.40/jobs`
`GET /slurmdb/v0.0.41/jobs`
* Ignore `--json` and `--yaml` options for `scontrol` show config
to prevent mixing output types.
* Fix not considering nodes in reservations with Maintenance or
Overlap flags when creating new reservations with `nodecnt` or
when they replace down nodes.
* Fix suspending/resuming steps running under a 23.02 `slurmstepd`
process.
* Fix options like `sprio --me` and `squeue --me` for users with
a uid greater than 2147483647.
* `fatal()` if `BlockSizes=0`. This value is invalid and would
otherwise cause the `slurmctld` to crash.
* `sacctmgr` - Fix issue where clearing out a preemption list using
`preempt=''` would cause the given qos to no longer be preempt-able
until set again.
* Fix `stepmgr` creating job steps concurrently.
* `data_parser/v0.0.40` - Avoid dumping "Infinity" for `NO_VAL` tagged
"number" fields.
* `data_parser/v0.0.41` - Avoid dumping "Infinity" for `NO_VAL` tagged
"number" fields.
* `slurmctld` - Fix a potential leak while updating a reservation.
* `slurmctld` - Fix state save with reservation flags when a update
fails.
* Fix reservation update issues with parameters Accounts and Users, when
using +/- signs.
* `slurmrestd` - Don't dump warning on empty wckeys in:
`GET /slurmdb/v0.0.40/config`
`GET /slurmdb/v0.0.41/config`
* Fix slurmd possibly leaving zombie processes on start up in configless
when the initial attempt to fetch the config fails.
* Fix crash when trying to drain a non-existing node (possibly deleted
before).
* `slurmctld` - fix segfault when calculating limit decay for jobs with
an invalid association.
* Fix IPMI energy gathering with multiple sensors.
* `data_parser/v0.0.39` - Remove xassert requiring errors and warnings
to have a source string.
* `slurmrestd` - Prevent potential segfault when there is an error
parsing an array field which could lead to a double xfree. This
applies to several endpoints in `data_parser` v0.0.39, v0.0.40 and
v0.0.41.
* `scancel` - Fix a regression from 23.11.6 where using both the
`--ctld` and `--sibling` options would cancel the federated job on
all clusters instead of only the cluster(s) specified by `--sibling`.
* `accounting_storage/mysql` - Fix bug when removing an association
specified with an empty partition.
* Fix setting multiple partition state restore on a job correctly.
* Fix difference in behavior when swapping partition order in job
submission.
* Fix security issue in stepmgr that could permit an attacker to
execute processes under other users' jobs. CVE-2024-48936.
* Wed Oct 23 2024 Egbert Eich <eich@suse.com>
- Add %(?%sysusers_requires} to slurm-config.
This fixes issues when building against Slurm.
* Mon Oct 14 2024 Egbert Eich <eich@suse.com>
- Update to version 24.05.3
* `data_parser/v0.0.40` - Added field descriptions.
* `slurmrestd` - Avoid creating new slurmdbd connection per request
to `* /slurm/slurmctld/*/*` endpoints.
* Fix compilation issue with `switch/hpe_slingshot` plugin.
* Fix gres per task allocation with threads-per-core.
* `data_parser/v0.0.41` - Added field descriptions.
* `slurmrestd` - Change back generated OpenAPI schema for
`DELETE /slurm/v0.0.40/jobs/` to `RequestBody` instead of using
parameters for request. `slurmrestd` will continue accept endpoint
requests via `RequestBody` or HTTP query.
* `topology/tree` - Fix issues with switch distance optimization.
* Fix potential segfault of secondary `slurmctld` when falling back
to the primary when running with a `JobComp` plugin.
* Enable `--json`/`--yaml=v0.0.39` options on client commands to
dump data using data_parser/v0.0.39 instead or outputting nothing.
* `switch/hpe_slingshot` - Fix issue that could result in a 0 length
state file.
* Fix unnecessary message protocol downgrade for unregistered nodes.
* Fix unnecessarily packing alias addrs when terminating jobs with
a mix of non-cloud/dynamic nodes and powered down cloud/dynamic
nodes.
* `accounting_storage/mysql` - Fix issue when deleting a qos that
could remove too many commas from the qos and/or delta_qos fields
of the assoc table.
* `slurmctld` - Fix memory leak when using RestrictedCoresPerGPU.
* Fix allowing access to reservations without `MaxStartDelay` set.
* Fix regression introduced in 24.05.0rc1 breaking
`srun --send-libs` parsing.
* Fix slurmd vsize memory leak when using job submission/allocation
commands that implicitly or explicitly use --get-user-env.
* `slurmd` - Fix node going into invalid state when using
`CPUSpecList` and setting CPUs to the # of cores on a
multithreaded node.
* Fix reboot asap nodes being considered in backfill after a restart.
* Fix `--clusters`/`-M queries` for clusters outside of a
federation when `fed_display` is configured.
* Fix `scontrol` allowing updating job with bad `cpus-per-task` value.
* `sattach` - Fix regression from 24.05.2 security fix leading to
crash.
* `mpi/pmix` - Fix assertion when built under `--enable-debug`.
- Changes from Slurm 24.05.2
* Fix energy gathering rpc counter underflow in
`_rpc_acct_gather_energy` when more than 10 threads try to get
energy at the same time. This prevented the possibility to get
energy from slurmd by any step until slurmd was restarted,
so losing energy accounting metrics in the node.
* `accounting_storage/mysql` - Fix issue where new user with `wckey`
did not have a default wckey sent to the slurmctld.
* `slurmrestd` - Prevent slurmrestd segfault when handling the
following endpoints when none of the optional parameters are
specified:
`DELETE /slurm/v0.0.40/jobs`
`DELETE /slurm/v0.0.41/jobs`
`GET /slurm/v0.0.40/shares`
`GET /slurm/v0.0.41/shares`
`GET /slurmdb/v0.0.40/instance`
`GET /slurmdb/v0.0.41/instance`
`GET /slurmdb/v0.0.40/instances`
`GET /slurmdb/v0.0.41/instances`
`POST /slurm/v0.0.40/job/{job_id}`
`POST /slurm/v0.0.41/job/{job_id}`
* Fix IPMI energy gathering when no IPMIPowerSensors are specified
in `acct_gather.conf`. This situation resulted in an accounted
energy of 0 for job steps.
* Fix a minor memory leak in slurmctld when updating a job dependency.
* `scontrol`,`squeue` - Fix regression that caused incorrect values
for multisocket nodes at `.jobs[].job_resources.nodes.allocation`
for `scontrol show jobs --(json|yaml)` and `squeue --(json|yaml)`.
* `slurmrestd` - Fix regression that caused incorrect values for
multisocket nodes at `.jobs[].job_resources.nodes.allocation` to
be dumped with endpoints:
`GET /slurm/v0.0.41/job/{job_id}`
`GET /slurm/v0.0.41/jobs`
* `jobcomp/filetxt` - Fix truncation of job record lines > 1024
characters.
* `switch/hpe_slingshot` - Drain node on failure to delete CXI
services.
* Fix a performance regression from 23.11.0 in CPU frequency
handling when no `CpuFreqDef` is defined.
* Fix one-task-per-sharing not working across multiple nodes.
* Fix inconsistent number of CPUs when creating a reservation
using the TRESPerNode option.
* `data_parser/v0.0.40+` - Fix job state parsing which could
break filtering.
* Prevent `cpus-per-task` to be modified in jobs where a `-c`
value has been explicitly specified and the requested memory
constraints implicitly increase the number of CPUs to allocate.
* `slurmrestd` - Fix regression where args `-s v0.0.39,dbv0.0.39`
and `-d v0.0.39` would result in `GET /openapi/v3` not
registering as a valid possible query resulting in 404 errors.
* `slurmrestd` - Fix memory leak for dbv0.0.39 jobs query which
occurred if the query parameters specified account, association,
cluster, constraints, format, groups, job_name, partition, qos,
reason, reservation, state, users, or wckey. This affects the
following endpoints:
`GET /slurmdb/v0.0.39/jobs`
* `slurmrestd` - In the case the slurmdbd does not respond to a
persistent connection init message, prevent the closed fd from
being used, and instead emit an error or warning depending on
if the connection was required.
* Fix 24.05.0 regression that caused the slurmdbd not to send back
an error message if there is an error initializing a persistent
connection.
* Reduce latency of forwarded x11 packets.
* Add `curr_dependency` (representing the current dependency of
the job).
and `orig_dependency` (representing the original requested
dependency of the job) fields to the job record in
`job_submit.lua` (for job update) and `jobcomp.lua`.
* Fix potential segfault of slurmctld configured with
`SlurmctldParameters=enable_rpc_queue` from happening on
reconfigure.
* Fix potential segfault of slurmctld on its shutdown when rate
limitting is enabled.
* `slurmrestd` - Fix missing job environment for `SLURM_JOB_NAME`,
`SLURM_OPEN_MODE`, `SLURM_JOB_DEPENDENCY`, `SLURM_PROFILE`,
`SLURM_ACCTG_FREQ`, `SLURM_NETWORK` and `SLURM_CPU_FREQ_REQ` to
match sbatch.
* Fix GRES environment variable indices being incorrect when only
using a subset of all GPUs on a node and the
`--gres-flags=allow-task-sharing` option.
* Prevent `scontrol` from segfaulting when requesting scontrol
show reservation `--json` or `--yaml` if there is an error
retrieving reservations from the `slurmctld`.
* `switch/hpe_slingshot` - Fix security issue around managing VNI
access. CVE-2024-42511.
* `switch/nvidia_imex` - Fix security issue managing IMEX channel
access. CVE-2024-42511.
* `switch/nvidia_imex` - Allow for compatibility with
`job_container/tmpfs`.
- Changes in Slurm 24.05.1
* Fix `slurmctld` and `slurmdbd` potentially stopping instead of
performing a logrotate when recieving `SIGUSR2` when using
`auth/slurm`.
* `switch/hpe_slingshot` - Fix slurmctld crash when upgrading
from 23.02.
* Fix "Could not find group" errors from `validate_group()` when
using `AllowGroups` with large `/etc/group` files.
* Add `AccountingStoreFlags=no_stdio` which allows to not record
the stdio paths of the job when set.
* `slurmrestd` - Prevent a slurmrestd segfault when parsing the
`crontab` field, which was never usable. Now it explicitly
ignores the value and emits a warning if it is used for the
following endpoints:
`POST /slurm/v0.0.39/job/{job_id}`
`POST /slurm/v0.0.39/job/submit`
`POST /slurm/v0.0.40/job/{job_id}`
`POST /slurm/v0.0.40/job/submit`
`POST /slurm/v0.0.41/job/{job_id}`
`POST /slurm/v0.0.41/job/submit`
`POST /slurm/v0.0.41/job/allocate`
* `mpi/pmi2` - Fix communication issue leading to task launch
failure with "`invalid kvs seq from node`".
* Fix getting user environment when using sbatch with
`--get-user-env` or `--export=` when there is a user profile
script that reads `/proc`.
* Prevent slurmd from crashing if `acct_gather_energy/gpu` is
configured but `GresTypes` is not configured.
* Do not log the following errors when `AcctGatherEnergyType`
plugins are used but a node does not have or cannot find sensors:
"`error: _get_joules_task: can't get info from slurmd`"
"`error: slurm_get_node_energy: Zero Bytes were transmitted or
received`"
However, the following error will continue to be logged:
"`error: Can't get energy data. No power sensors are available.
Try later`"
* `sbatch`, `srun` - Set `SLURM_NETWORK` environment variable if
`--network` is set.
* Fix cloud nodes not being able to forward to nodes that restarted
with new IP addresses.
* Fix cwd not being set correctly when running a SPANK plugin with a
`spank_user_init()` hook and the new "`contain_spank`" option set.
* `slurmctld` - Avoid deadlock during shutdown when `auth/slurm`
is active.
* Fix segfault in `slurmctld` with `topology/block`.
* `sacct` - Fix printing of job group for job steps.
* `scrun` - Log when an invalid environment variable causes the
job submission to be rejected.
* `accounting_storage/mysql` - Fix problem where listing or
modifying an association when specifying a qos list could hang
or take a very long time.
* `gpu/nvml` - Fix `gpuutil/gpumem` only tracking last GPU in step.
Now, `gpuutil/gpumem` will record sums of all GPUS in the step.
* Fix error in `scrontab` jobs when using
`slurm.conf:PropagatePrioProcess=1`.
* Fix `slurmctld` crash on a batch job submission with
`--nodes 0,...`.
* Fix dynamic IP address fanout forwarding when using `auth/slurm`.
* Restrict listening sockets in the `mpi/pmix` plugin and `sattach`
to the `SrunPortRange`.
* `slurmrestd` - Limit mime types returned from query to
`GET /openapi/v3` to only return one mime type per serializer
plugin to fix issues with OpenAPI client generators that are
unable to handle multiple mime type aliases.
* Fix many commands possibly reporting an "`Unexpected Message
Received`" when in reality the connection timed out.
* Prevent slurmctld from starting if there is not a json
serializer present and the `extra_constraints` feature is enabled.
* Fix heterogeneous job components not being signaled with
`scancel --ctld` and `DELETE slurm/v0.0.40/jobs` if the job ids
are not explicitly given, the heterogeneous job components match
the given filters, and the heterogeneous job leader does not
match the given filters.
* Fix regression from 23.02 impeding job licenses from being cleared.
* Move error to `log_flag` which made `_get_joules_task` error to
be logged to the user when too many rpcs were queued in slurmd
for gathering energy.
* For `scancel --ctld` and the associated rest api endpoints:
`DELETE /slurm/v0.0.40/jobs`
`DELETE /slurm/v0.0.41/jobs`
Fix canceling the final array task in a job array when the task
is pending and all array tasks have been split into separate job
records. Previously this task was not canceled.
* Fix `power_save operation` after recovering from a failed
reconfigure.
* `slurmctld` - Skip removing the pidfile when running under
systemd. In that situation it is never created in the first place.
* Fix issue where altering the flags on a Slurm account
(`UsersAreCoords`) several limits on the account's association
would be set to 0 in Slurm's internal cache.
* Fix memory leak in the controller when relaying `stepmgr` step
accounting to the dbd.
* Fix segfault when submitting stepmgr jobs within an existing
allocation.
* Added `disable_slurm_hydra_bootstrap` as a possible `MpiParams`
parameter in `slurm.conf`. Using this will disable env variable
injection to allocations for the following variables:
`I_MPI_HYDRA_BOOTSTRAP,` `I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS`,
`HYDRA_BOOTSTRAP`, `HYDRA_LAUNCHER_EXTRA_ARGS`.
* `scrun` - Delay shutdown until after start requested.
This caused `scrun` to never start or shutdown and hung forever
when using `--tty`.
* Fix backup `slurmctld` potentially not running the agent when
taking over as the primary controller.
* Fix primary controller not running the agent when a reconfigure
of the `slurmctld` fails.
* `slurmd` - fix premature timeout waiting for
`REQUEST_LAUNCH_PROLOG` with large array jobs causing node to
drain.
* `jobcomp/{elasticsearch,kafka}` - Avoid sending fields with
invalid date/time.
* `jobcomp/elasticsearch` - Fix `slurmctld` memory leak from
curl usage.
* `acct_gather_profile/influxdb` - Fix slurmstepd memory leak from
curl usage
* Fix 24.05.0 regression not deleting job hash dirs after
`MinJobAge`.
* Fix filtering arguments being ignored when using squeue `--json`.
* `switch/nvidia_imex` - Move setup call after `spank_init()` to
allow namespace manipulation within the SPANK plugin.
* `switch/nvidia_imex` - Skip plugin operation if
`nvidia-caps-imex-channels` device is not present rather than
preventing slurmd from starting.
* `switch/nvidia_imex` - Skip plugin operation if
`job_container/tmpfs` is configured due to incompatibility.
* `switch/nvidia_imex` - Remove any pre-existing channels when
`slurmd` starts.
* `rpc_queue` - Add support for an optional `rpc_queue.yaml`
configuration file.
* `slurmrestd` - Add new +prefer_refs flag to `data_parser/v0.0.41`
plugin. This flag will avoid inlining single referenced schemas
in the OpenAPI schema.
* Tue Jun 04 2024 Christian Goll <cgoll@suse.com>
- Updated to new release 24.05.0 with following major changes
* Important Notes:
If using the slurmdbd (Slurm DataBase Daemon) you must update
this first. NOTE: If using a backup DBD you must start the
primary first to do any database conversion, the backup will not
start until this has happened. The 24.05 slurmdbd will work
with Slurm daemons of version 23.02 and above. You will not
need to update all clusters at the same time, but it is very
important to update slurmdbd first and having it running before
updating any other clusters making use of it.
* Highlights
+ Federation - allow client command operation when slurmdbd is
unavailable.
+ `burst_buffer/lua` - Added two new hooks: `slurm_bb_test_data_in`
and `slurm_bb_test_data_out`. The syntax and use of the new hooks
are documented in `etc/burst_buffer.lua.example`. These are
required to exist. slurmctld now checks on startup if the
`burst_buffer.lua` script loads and contains all required hooks;
`slurmctld` will exit with a fatal error if this is not
successful. Added `PollInterval` to `burst_buffer.conf`. Removed
the arbitrary limit of 512 copies of the script running
simultaneously.
+ Add QOS limit `MaxTRESRunMinsPerAccount`.
+ Add QOS limit `MaxTRESRunMinsPerUser`.
+ Add `ELIGIBLE` environment variable to `jobcomp/script` plugin.
+ Always use the QOS name for `SLURM_JOB_QOS` environment variables.
Previously the batch environment would use the description field,
which was usually equivalent to the name.
+ `cgroup/v2` - Require dbus-1 version >= 1.11.16.
+ Allow `NodeSet` names to be used in SuspendExcNodes.
+ `SuspendExcNodes=<nodes>:N` now counts allocated nodes in `N`.
The first `N` powered up nodes in <nodes> are protected from
being suspended.
+ Store job output, input and error paths in `SlurmDBD`.
+ Add `USER_DELETE` reservation flag to allow users with access
to a reservation to delete it.
+ Add `SlurmctldParameters=enable_stepmgr` to enable step
management through the `slurmstepd` instead of the controller.
+ Added `PrologFlags=RunInJob` to make prolog and epilog run
inside the job extern step to include it in the job's cgroup.
+ Add ability to reserve MPI ports at the job level for stepmgr
jobs and subdivide them at the step level.
+ `slurmrestd` - Add `--generate-openapi-spec argument`.
* Configuration File Changes (see appropriate man page for details)
+ `CoreSpecPlugin` has been removed.
+ Removed `TopologyPlugin` tree and dragonfly support from
`select/linear`. If those topology plugins are desired please
switch to `select/cons_tres`.
+ Changed the default value for `UnkillableStepTimeout` to 60
seconds or five times the value of `MessageTimeout`, whichever
is greater.
+ An error log has been added if `JobAcctGatherParams` '`UsePss`'
or '`NoShare`' are configured with a plugin other than
`jobacct_gather/linux`. In such case these parameters are ignored.
+ `helpers.conf` - Added `Flags=rebootless` parameter allowing
feature changes without rebooting compute nodes.
+ `topology/block` - Replaced the `BlockLevels` with `BlockSizes`
in `topology.conf`.
+ Add `contain_spank` option to `SlurmdParameters`. When set,
`spank_user_init()`, `spank_task_post_fork()`, and
`spank_task_exit()` will execute within the
`job_container/tmpfs` plugin namespace.
+ Add `SlurmctldParameters=max_powered_nodes=N`, which prevents
powering up nodes after the max is reached.
+ Add `ExclusiveTopo` to a partition definition in `slurm.conf`.
+ Add `AccountingStorageParameters=max_step_records` to limit how
many steps are recorded in the database for each job - excluding
batch.
* Command Changes (see man pages for details)
+ Add support for "elevenses" as an additional time specification.
+ Add support for `sbcast --preserve` when `job_container/tmpfs`
configured (previously documented as unsupported).
+ `scontrol` - Add new subcommand `power` for node power control.
+ `squeue` - Adjust `StdErr`, `StdOut`, and `StdIn` output formats.
These will now consistently print "`(null)`" if a value is
unavailable. `StdErr` will no longer display `StdOut` if it is
not distinctly set. `StdOut` will now correctly display the
default filename pattern for job arrays, and no longer show it
for non-batch jobs. However, the expansion patterns will
no longer be substituted by default.
+ Add `--segment` to job allocation to be used in topology/block.
+ Add `--exclusive=topo` for use with topology/block.
+ `squeue` - Add `--expand-patterns` option to expand `StdErr`,
`StdOut`, `StdIn` filename patterns as best as possible.
+ `sacct` - Add `--expand-patterns` option to expand `StdErr`,
`StdOut`, `StdIn` filename patterns as best as possible.
+ `sreport` - Requesting `format=Planned` will now return the
expected `Planned` time as documented, instead of `PlannedDown`.
To request `Planned Down`, one must use now `format=PLNDDown`
or `format=PlannedDown` explicitly. The abbreviations
"`Pl`" or "`Pla`" will now make reference to Planned instead
of `PlannedDown`.
* API Changes
+ Removed `ListIterator` type from `<slurm/slurm.h>`.
+ Removed `slurm_xlate_job_id()` from `<slurm/slurm.h>`
* SLURMRESTD Changes
+ `openapi/dbv0.0.38` and `openapi/v0.0.38` plugins have been
removed.
+ `openapi/dbv0.0.39` and `openapi/v0.0.39` plugins have been
tagged as deprecated to warn of their removal in the next release.
+ Changed `slurmrestd.service` to only listen on TCP socket by
default. Environments with existing drop-in units for the
service may need further adjustments to work after upgrading.
+ `slurmrestd` - Tagged `script` field as deprecated in
`POST /slurm/v0.0.41/job/submit` in anticipation of removal in
future OpenAPI plugin versions. Job submissions should set the
`job.script` (or `jobs[0].script` for HetJobs) fields instead.
+ `slurmrestd` - Attempt to automatically convert enumerated
string arrays with incoming non-string values into strings.
Add warning when incoming value for enumerated string arrays
can not be converted to string and silently ignore instead of
rejecting entire request. This change affects any endpoint that
uses an enunmerated string as given in the OpenAPI specification.
An example of this conversion would be to
`POST /slurm/v0.0.41/job/submit` with `.job.exclusive = true`.
While the JSON (boolean) true value matches a possible
enumeration, it is not the expected "true" string. This change
automatically converts the (boolean) `true` to (string) "`true`"
avoiding a parsing failure.
+ `slurmrestd` - Add `POST /slurm/v0.0.41/job/allocate` endpoint.
This endpoint will create a new job allocation without any steps.
The allocation will need to be ended via signaling the job or
it will run to the timelimit.
+ `slurmrestd` - Allow startup when `slurmdbd` is not configured
and avoid loading `slurmdbd` specific plugins.
* MPI/PMI2 Changes
+ Jobs submitted with the `SLURM_HOSTFILE` environment variable
set implies using an arbitrary distribution. Nevertheless, the
logic used in PMI2 when generating their associated
`PMI_process_mapping` values has been changed and will now be
the same used for the plane distribution, as if `-m plane` were
used. This has been changed because the original arbitrary
distribution implementation did not account for multiple
instances of the same host being present in `SLURM_HOSTFILE`,
providing an incorrect process mapping in such case. This
change also enables distributing tasks in blocks when using
arbitrary distribution, which was not the case before. This
only affects `mpi`/`pmi2` plugin.
- Removed Fix-test-21.41.patch as upstream test changed.
- Dropped package plugin-ext-sensors-rrd as the plugin module no
longer exists.
* Mon Mar 25 2024 Christian Goll <cgoll@suse.com>
- removed Keep-logs-of-skipped-test-when-running-test-cases-sequentially.patch
as incoperated upstream
- Changes in Slurm 23.02.5
* Add the `JobId` to `debug()` messages indicating when
`cpus_per_task/mem_per_cpu` or `pn_min_cpus` are being
automatically adjusted.
* Fix regression in 23.02.2 that caused `slurmctld -R` to crash on
startup if a node features plugin is configured.
* Fix and prevent reoccurring reservations from overlapping.
* `job_container/tmpfs` - Avoid attempts to share `BasePath`
between nodes.
* Change the log message warning for rate limited users from
verbose to info.
* With `CR_Cpu_Memory`, fix node selection for jobs that request
gres and `--mem-per-cpu`.
* Fix a regression from 22.05.7 in which some jobs were allocated
too few nodes, thus overcommitting cpus to some tasks.
* Fix a job being stuck in the completing state if the job ends
while the primary controller is down or unresponsive and the
backup controller has not yet taken over.
* Fix `slurmctld` segfault when a node registers with a configured
`CpuSpecList` while slurmctld configuration has the node without
`CpuSpecList`.
* Fix cloud nodes getting stuck in `POWERED_DOWN+NO_RESPOND` state
after not registering by `ResumeTimeout`.
* `slurmstepd` - Avoid cleanup of `config.json`-less containers
spooldir getting skipped.
* `slurmstepd` - Cleanup per task generated environment for
containers in spooldir.
* Fix `scontrol segfault` when 'completing' command requested
repeatedly in interactive mode.
* Properly handle a race condition between `bind()` and `listen()`
calls in the network stack when running with `SrunPortRange` set.
* Federation - Fix revoked jobs being returned regardless of the
`-a`/`--all` option for privileged users.
* Federation - Fix canceling pending federated jobs from non-origin
clusters which could leave federated jobs orphaned from the origin
cluster.
* Fix sinfo segfault when printing multiple clusters with
`--noheader` option.
* Federation - fix clusters not syncing if clusters are added to
a federation before they have registered with the dbd.
* Change `pmi2` plugin to honor the `SrunPortRange` option. This
matches the new behavior of the pmix plugin in 23.02.0. Note that
neither of these plugins makes use of the "`MpiParams=ports=`"
option, and previously were only limited by the systems ephemeral
port range.
* `node_features/helpers` - Fix node selection for jobs requesting
changeable features with the '`|`' operator, which could prevent
jobs from running on some valid nodes.
* `node_features/helpers` - Fix inconsistent handling of '`&`' and
'`|`', where an AND'd feature was sometimes AND'd to all sets of
features instead of just the current set. E.g. "`foo|bar&baz`" was
interpreted as `{foo,baz}` or `{bar,baz}` instead of how it is
documented: "`{foo} or {bar,baz}`".
* Fix job accounting so that when a job is requeued its allocated
node count is cleared. After the requeue, sacct will correctly
show that the job has 0 `AllocNodes` while it is pending or if
it is canceled before restarting.
* `sacct` - `AllocCPUS` now correctly shows 0 if a job has not yet
received an allocation or if the job was canceled before getting
one.
* Fix intel oneapi autodetect: detect the `/dev/dri/renderD[0-9]+`
gpus, and do not detect `/dev/dri/card[0-9]+`.
* Format batch, extern, interactive, and pending step ids into
strings that are human readable.
* Fix node selection for jobs that request `--gpus` and a number
of tasks fewer than gpus, which resulted in incorrectly rejecting
these jobs.
* Remove `MYSQL_OPT_RECONNECT` completely.
* Fix cloud nodes in `POWERING_UP` state disappearing (getting set
to `FUTURE`) when an `scontrol reconfigure` happens.
* `openapi/dbv0.0.39` - Avoid assert / segfault on missing
coordinators list.
* `slurmrestd` - Correct memory leak while parsing OpenAPI
specification templates with server overrides.
* `slurmrestd` - Reduce memory usage when printing out job CPU
frequency.
* Fix overwriting user node reason with system message.
* Remove `--uid` / `--gid` options from salloc and srun commands.
* Prevent deadlock when rpc_queue is enabled.
* `slurmrestd` - Correct OpenAPI specification generation bug where
fields with overlapping parent paths would not get generated.
* Fix memory leak as a result of a partition info query.
* Fix memory leak as a result of a job info query.
* slurmrestd - For `GET /slurm/v0.0.39/node[s]`, change format of
node's energy field `current_watts` to a dictionary to account
for unset value instead of dumping `4294967294`.
* `slurmrestd` - For `GET /slurm/v0.0.39/qos`, change format of
QOS's field `priority` to a dictionary to account for unset
value instead of dumping `4294967294`.
* `slurmrestd` - For `GET /slurm/v0.0.39/job[s]`, the `return code`
code field in `v0.0.39_job_exit_code` will be set to 127 instead
of being left unset where job does not have a relevant return code.
* `data_parser/v0.0.39` - Add `required/memory_per_cpu` and
required/memory_per_node to `sacct --json` and `sacct --yaml` and
`GET /slurmdb/v0.0.39/jobs` from `slurmrestd`.
* For step allocations, fix `--gres=none` sometimes not ignoring
gres from the job.
* Fix `--exclusive` jobs incorrectly gang-scheduling where they
shouldn't.
* Fix allocations with `CR_SOCKET`, gres not assigned to a specific
socket, and block core distribion potentially allocating more
sockets than required.
* `gpu/oneapi` - Store cores correctly so CPU affinity is tracked.
* Revert a change in 23.02.3 where Slurm would kill a script's
process group as soon as the script ended instead of waiting as
long as any process in
that process group held the stdout/stderr file descriptors open.
That change broke some scripts that relied on the previous
behavior. Setting time limits for scripts (such as
`PrologEpilogTimeout`) is strongly encouraged to avoid Slurm
waiting indefinitely for scripts to finish.
* Allow slurmdbd -R to work if the root assoc id is not 1.
* Fix `slurmdbd -R` not returning an error under certain conditions.
* `slurmdbd` - Avoid potential NULL pointer dereference in the
mysql plugin.
* Revert a change in 23.02 where `SLURM_NTASKS` was no longer
set in the job's environment when `--ntasks-per-node` was
requested.
* Limit periodic node registrations to 50 instead of the full
`TreeWidth`.
Since unresolvable `cloud/dynamic` nodes must disable fanout by
setting `TreeWidth` to a large number, this would cause all nodes
to register at once.
* Fix regression in 23.02.3 which broken x11 forwarding for hosts
when `MUNGE` sends a localhost address in the encode host field.
This is caused when the node hostname is mapped to 127.0.0.1
(or similar) in `/etc/hosts`.
* `openapi/[db]v0.0.39` - fix memory leak on parsing error.
* `data_parser/v0.0.39` - fix updating qos for associations.
* `openapi/dbv0.0.39` - fix updating values for associations with
null users.
* Fix minor memory leak with `--tres-per-task` and licenses.
* Fix cyclic socket cpu distribution for tasks in a step where
`--cpus-per-task` < usable threads per core.
- Changes in Slurm 23.02.4
* Fix `sbatch` return code when --wait is requested on a job array.
* `switch/hpe_slingshot` - avoid segfault when running with old
libcxi.
* Avoid slurmctld segfault when specifying
`AccountingStorageExternalHost`.
* Fix collected `GPUUtilization` values for `acct_gather_profile`
plugins.
* Fix slurmrestd handling of job hold/release operations.
* Make spank `S_JOB_ARGV` item value hold the requested command
argv instead of the srun `--bcast` value when `--bcast` requested
(only in local context).
* Fix step running indefinitely when slurmctld takes more than
`MessageTimeout` to respond. Now, `slurmctld` will cancel the
step when detected, preventing following steps from getting stuck
waiting for resources to be released.
* Fix regression to make job_desc.min_cpus accurate again in
job_submit when requesting a job with `--ntasks-per-node`.
* `scontrol` - Permit changes to `StdErr` and `StdIn` for pending
jobs.
* `scontrol` - Reset std{err,in,out} when set to empty string.
* `slurmrestd` - mark environment as a required field for job
submission descriptions.
* `slurmrestd` - avoid dumping null in OpenAPI schema required
fields.
`data_parser/v0.0.39` - avoid rejecting valid `memory_per_node`
formatted as dictionary provided with a job description.
* `data_parser/v0.0.39` - avoid rejecting valid `memory_per_cpu`
formatted as dictionary provided with a job description.
* `slurmrestd` - Return HTTP error code 404 when job query fails.
* `slurmrestd` - Add return schema to error response to job and
license query.
* Fix handling of ArrayTaskThrottle in backfill.
* Fix regression in 23.02.2 when checking gres state on `slurmctld`
startup or reconfigure. Gres changes in the configuration were
not updated on `slurmctld` startup. On startup or reconfigure,
these messages were present in the log:
"`error: Attempt to change gres/gpu Count`".
* Fix potential double count of gres when dealing with limits.
* `switch/hpe_slingshot` - support alternate traffic class names
with "`TC_`" prefix.
* `scrontab` - Fix cutting off the final character of quoted
variables.
* Fix `slurmstepd` segfault when `ContainerPath` is not set in
`oci.conf`.
* Change the log message warning for rate limited users from
debug to verbose.
* Fixed an issue where jobs requesting licenses were incorrectly
rejected.
* `smail` - Fix issues where emails at job completion were not
being sent.
* `scontrol/slurmctld` - fix comma parsing when updating a
reservation's nodes.
* `cgroup/v2` - Avoid capturing log output for ebpf when
constraining devices, as this can lead to inadvertent failure
if the log buffer is too small.
* Fix --gpu-bind=single binding tasks to wrong gpus, leading to
some gpus having more tasks than they should and other gpus being
unused.
* Fix main scheduler loop not starting after failover to backup
controller.
* Added error message when attempting to use sattach on batch or
extern steps.
* Fix regression in 23.02 that causes slurmstepd to crash when
`srun` requests more than `TreeWidth` nodes in a step and uses
the `pmi2` or `pmix` plugin.
* Reject job `ArrayTaskThrottle` update requests from unprivileged
users.
* `data_parser/v0.0.39` - populate description fields of property
objects in generated OpenAPI specifications where defined.
* `slurmstepd` - Avoid segfault caused by ContainerPath not being
terminated by '`/`' in `oci.conf`.
* `data_parser/v0.0.39` - Change `v0.0.39_job_info` response to tag
`exit_code` field as being complex instead of only an unsigned
integer.
* `job_container/tmpfs` - Fix %h and %n substitution in `BasePath`
where `%h` was substituted as the `NodeName` instead of the
hostname, and `%n` was substituted as an empty string.
* Fix regression where --cpu-bind=verbose would override
`TaskPluginParam`.
* `scancel` - Fix `--clusters`/`-M` for federations. Only filtered
jobs (e.g. -A, -u, -p, etc.) from the specified clusters will be
canceled, rather than all jobs in the federation.
Specific jobids will still be routed to the origin cluster
for cancellation.
* Mon Jan 29 2024 Egbert Eich <eich@suse.com>
- Update to version 23.11.03
* slurmrestd - Reject single http query with multiple path
requests.
* Fix launching Singularity v4.x containers with
`srun --container` by setting .process.terminal to true in
generated `config.json` when step has pseudoterminal (`--pty`)
requested.
* Fix loading in `dyanmic/cloud` node jobs after `net_cred`
expired.
* Fix cgroup null path error on `slurmd/slurmstepd` tear down.
* `data_parser/v0.0.40` - Prevent failure if accounting is
disabled, instead issue a warning if needed data from the
database can not be retrieved.
* `openapi/slurmctld` - Prevent failure if accounting is disabled.
* Prevent `slurmscriptd` processing delays from blocking other
threads in `slurmctld` while trying to launch various scripts.
This is additional work for a fix in 23.02.6.
* Fix memory leak when receiving alias addrs from controller.
* `scontrol` - Accept `scontrol token lifespan=infinite` to
create tokens that effectively do not expire.
* Avoid errors when Slurmdb accounting disabled when `--json` or
`--yaml` is invoked with CLI commands and `slurmrestd`. Add
warnings when query would have populated data from Slurmdb
instead of errors.
* Fix `slurmctld` memory leak when running job with
`--tres-per-task=gres:shard:#`
* Fix backfill trying to start jobs outside of backfill window.
* Fix oversubscription on partitions with `PreemptMode=OFF`.
* Preserve node reason on power up if the node is downed
or drained.
* `data_parser/v0.0.40` - Avoid aborting when invoking a not
implemented parser.
* `data_parser/v0.0.40` - Fix how nice values are parsed for job
submissions.
* `data_parser/v0.0.40` - Fix regression where parsing error did
not result in invalid request being rejected.
* Fix segfault in front-end node registration.
* Prevent jobs using none typed gpus from being killed by the
controller after a reconfig or restart.
* Fix deadlock situation in the dbd when adding associations.
* Update default values of text/blob columns when updating from
old mysql versions in more situations. This improves a
previous fix to handle an uncommon case when upgrading
mysql/mariadb.
* Fix rpmbuild in openSUSE/SLES due to incorrect mariadb
dependency.
* When upgrading the slurmdbd to 23.11, avoid generating a query
to update the association table that is larger than
`max_allowed_packet` which would result in an upgrade failure.
* Fix rare deadlock when a dynamic node registers at the same
time that a once per minute background task occurs.
* `data_parser/v0.0.40` - Fix enumerated strings in OpenAPI
specification not have type field specified.
* Improve `scontrol show job -d` information of used shared
gres (`shard/mps`) topology.
* accounting_storage/mysql - Fix usage query to use new lineage
column instead of lft/rgt.
* `slurmrestd` - Improve handling of missing parsers when
content plugins expect parsers not loaded.
* `slurmrestd` - Correct parsing of StepIds when querying jobs.
* `slurmrestd` - Improve error from parsing failures of lists.
* `slurmrestd` - Improve parsing of singular values for lists.
* `accounting_storage/mysql` - Fix `PrivateData=User` when
listing associations.
* Disable sorting of dynamic nodes to avoid issues when
restarting with heterogenous jobs that cause jobs to abort on
restart.
* Don't allow deletion of non-dynamic nodes.
* `accounting_storage/mysql` - Fix issue adding partition based
associations.
* Respect non-"slurm" settings for `I_MPI_HYDRA_BOOTSTRAP` and
`HYDRA_BOOTSTRAP` and avoid injecting the `--external-launcher`
option which will cause `mpirun/mpiexec` to fail with an
unexpected argument error.
* Fix bug where scontrol hold would change node count for jobs
with implicitly defined node counts.
* `data_parser/v0.0.40` - Fix regression of support for "hold"
in job description.
* Avoid sending KILL RPCs to unresolvable `POWERING_UP` and
`POWERED_DOWN` nodes.
* `data_parser/v0.0.38` - Fix several potential NULL
dereferences that could cause slurmrestd to crash.
* Add `--gres-flags=one-task-per-sharing`. Do not allow different
tasks in to be allocated shared gres from the same sharing gres.
* Add `SelectTypeParameters=ENFORCE_BINDING_GRES` and
`ONE_TASK_PER_SHARING_GRES`.
This gives default behavior for a job's `--gres-flags`.
* Alter the networking code to try connecting to the backup
controllers if the DNS lookup for the primary `SlurmctldHost`
fails.
* Alter the name resolution to only log at `verbose()` in client
commands on failures. This allows for HA setups where the DNS
entries are withdrawn for some `SlurmctldHost` entries without
flooding the user with errors.
* Prevent `slurmscriptd` PID leaks when running `slurmctld` in
foreground mode.
* Open all `slurmctld` listening ports at startup, and persist
throughout.
This also changes the backup `slurmctld` process to open the
`SlurmctldPort` range, instead of only the first.
* Fix backup `slurmctld` shutting down instead of resuming
standby duty if it took control.
* Fix race condition that delayed the primary `slurmctld`
resuming when taking control from a backup controller.
* `srun` - Ensure processed messages are meant for this job in
case of a rapidly-reused TCP port.
* `srun` - Prevent step launch failure while waiting for step
allocation if a stray message is received.
* Fix backup `slurmctld` to be able to respond to configless
config file requests correctly.
* Fix `slurmctld` crashing when recovering from a failed
reconfigure.
* Fix `slurmscriptd` operation after recovering from a failed
reconfigure.
- Make sure `-std=gnu99` is added to CFLAGS on SLE-12.
- Use %%autopatch.
* Fri Jan 12 2024 Christian Goll <cgoll@suse.com>
- Update to 23.11.1 with following major improvements and fixing
CVE-2023-49933, CVE-2023-49934, CVE-2023-49935, CVE-2023-49936
and CVE-2023-49937
* Substantially overhauled the SlurmDBD association management
code. For clusters updated to 23.11, account and user
additions or removals are significantly faster than in prior
releases.
* Overhauled `scontrol reconfigure` to prevent configuration
mistakes from disabling slurmctld and slurmd. Instead, an
error will be returned, and the running configuration will
persist. This does require updates to the systemd service
files to use the `--systemd` option to `slurmctld` and `slurmd`.
* Added a new internal `auth/cred` plugin - `auth/slurm`. This
builds off the prior `auth/jwt` model, and permits operation
of the `slurmdbd` and `slurmctld` without access to full
directory information with a suitable configuration.
* Added a new `--external-launcher` option to `srun`, which is
automatically set by common MPI launcher implementations and
ensures processes using those non-srun launchers have full
access to all resources allocated on each node.
* Reworked the dynamic/cloud modes of operation to allow for
"fanout" - where Slurm communication can be automatically
offloaded to compute nodes for increased cluster scalability.
* Overhauled and extended the Reservation subsystem to allow
for most of the same resource requirements as are placed on
the job. Notably, this permits reservations to now reserve
GRES directly.
- Details of changes:
* Fix `scontrol update job=... TimeLimit+=/-=` when used with a
raw JobId of job array element.
* Reject `TimeLimit` increment/decrement when called on job with
`TimeLimit=UNLIMITED`.
* Fix issue with requesting a job with `*licenses` as well as
`*tres-per-task=license`.
* `slurmctld` - Prevent segfault in `getopt_long()` with an
invalid long option.
* slurmrestd - Added `/meta/slurm/cluster` field to responses.
* Adjust systemd service files to start daemons after
`remote-fs.target`.
* Fix `task/cgroup` indexing tasks in cgroup plugins, which
caused `jobacct/gather` to match the gathered stats with the
wrong task id.
* `select/linear` - Fix regression in 23.11 in which jobs that
requested `*cpus-per-task` were rejected.
* `data_parser/v0.0.40` - Fix the parsing for
`/slurmdb/v0.0.40/jobs` exit_code query parameter.
* If a job requests more shards which would allocate more than
one sharing GRES (gpu) per node refuse it unless
`SelectTypeparameters` has `MULTIPLE_SHARING_GRES_PJ`.
* Trigger fatal exit when Slurm API function is called before
`slurm_init()` is called.
* `slurmd` - Fix issue with `scontrol reconfigure` when started
with `-c`.
* `slurmrestd` - Job submissions that result in the following
error codes will be considered as successfully submitted (with
a warning), instead of returning an HTTP 500 error back:
`ESLURM_NODES_BUSY`, `ESLURM_RESERVATION_BUSY`, `ESLURM_JOB_HELD`,
`ESLURM_NODE_NOT_AVAIL`, `ESLURM_QOS_THRES`,
`ESLURM_ACCOUNTING_POLICY`, `ESLURM_RESERVATION_NOT_USABLE`,
`ESLURM_REQUESTED_PART_CONFIG_UNAVAILABLE`,
`ESLURM_BURST_BUFFER_WAIT`, ESLURM_PARTITION_DOWN`,
`ESLURM_LICENSES_UNAVAILABLE`.
* Fix a `slurmctld` fatal error when upgrading to 23.11 and
changing from `select/cons_res` to `select/cons_tres` at the
same time.
* `slurmctld` - Reject arbitrary distribution jobs that have a
minimum node count that differs from the number of unique
nodes in the hostlist.
* Prevent `slurmdbd` errors when updating reservations with names
containing apostrophes.
* Prevent message extension attacks that could bypass the
message hash. CVE-2023-49933.
* Prevent SQL injection attacks in `slurmdbd`. CVE-2023-49934.
* Prevent message hash bypass in slurmd which can allow an
attacker to reuse root-level MUNGE tokens and escalate
permissions. CVE-2023-49935.
* Prevent NULL pointer dereference on size_valp overflow.
CVE-2023-49936.
* Prevent double-xfree() on error in `_unpack_node_reg_resp()`.
CVE-2023-49937.
* For jobs that request `*cpus-per-gpu`, ensure that the
`*cpus-per-gpu request` is honored on every node in the and
not just for the job as a whole.
* Fix listing available `data_parser` plugins for json and yaml
when giving no commands to `scontrol` or `sacctmgr`.
* `slurmctld` - Rework `scontrol reconfigure` to avoid race
conditions that can result in stray jobs.
* `slurmctld` - Shave ~1 second off average reconfigure time by
terminating internal processing threads faster.
* Skip running `slurmdbd -R` if the connected cluster is 23.11
or newer. This operation is no longer relevant for 23.11.
* Ensure `slurmscriptd` shuts down before `slurmctld` is stopped
or reconfigured.
* Improve error handling and error messages in `slurmctld` to
`slurmscriptd` communications. This includes avoiding
potential deadlock in `slurmctld` if slurmscript dies
unexpectedly.
* Do not hold batch jobs whose extra constraints cannot be
immediately satisfied, and set the state reason to
`Constraints` instead of `BadConstraints`.
* Fix verbose log message printing a hex number instead of a job
id.
* Upgrade rate limit parameters message from debug to info.
* For `SchedulerParameters=extra_constraints`, prevent `slurmctld`
segfault when starting a `slurmd` with `*extra` for a node
that did not previously set this.
This also ensures the extra constraints model works off the
current node state, not the prior state.
* Fix `*tres-per-task` assertion.
* Fix a few issues when creating reservations.
* Add `SchedulerParameters=time_min_as_soft_limit` option.
* Remove `SLURM_WORKING_CLUSTER` env from batch and srun
environments.
* `cli_filter/lua` - return nil for unset time options rather
than the string `2982616-04:14:00` (which is the internal
macro `NO_VAL` represented as time string).
* Remove 'none' plugins for all but auth and cred. scontrol show
config will report (null) now.
* Removed `select/cons_res`. Please update your configuration to
`select/cons_tres`.
* `mpi/pmix` - When aborted with status 0, avoid marking
job/step as failed.
* Fixed typo on `initialized` for the description of
`ESLURM_PLUGIN_NOT_LOADED`.
* `cgroup.conf` - Removed deprecated parameters `AllowedKmemSpace`,
`ConstrainKmemSpace`, `MaxKmemPercent`, and `MinKmemSpace`.
* `proctrack/cgroup` - Add `SignalChildrenProcesses=<yes|no>`
option to `cgroup.conf`. This allows signals for cancelling,
suspending, resuming, etc. to be sent to children processes in
a `step/job` rather than just the parent.
* Add `PreemptParameters=suspend_grace_time` parameter to
control amount of time between `SIGTSTP` and `SIGSTOP` signals
when suspending jobs.
* `job_submit/throttle` - improve reset of submitted job counts
per user in order to better honor
`SchedulerParameters=jobs_per_user_per_hour=#`.
* Load the user environment into a private pid namespace to
avoid user scripts leaving background processes on a node.
* `scontrol` show `assoc_mgr` will display Lineage instead of Lft
for associations.
* Add `SlurmctldParameters=no_quick_restart` to avoid a new
`slurmctld` taking over the old `slurmctld` by accident.
* Fix `--cpus-per-gpu` for step allocations, which was
previously ignored for job steps. `*cpus-per-gpu` implies
`--exact`.
* Fix mutual exclusivity of `--cpus-per-gpu` and
`--cpus-per-task`: fatal if both options are requested in the
commandline or both are requested in the environment. If one
option is requested in the command line, it will override the
other option in the environment.
* `slurmrestd` - `openapi/dbv0.0.37` and `openapi/v0.0.37`
plugins have been removed.
* `slurmrestd` - `openapi/dbv0.0.38` and `openapi/v0.0.38`
plugins have been tagged as deprecated.
* `slurmrestd` - added auto population of `info/version` field.
* `sdiag` - add `--yaml` and `--json` arg support to specify
data_parser plugin.
* `sacct` - add `--yaml` and `--json` arg support to specify
`data_parser` plugin.
* `scontrol` - add `--yaml` and `--json` arg support to specify
`data_parser` plugin.
* `sinfo` - add `--yaml` and `--json` arg support to specify
`data_parser` plugin.
* `squeue` - add `--yaml` and `--json` arg support to specify
`data_parser` plugin.
* Changed the default `SelectType` to `select/cons_tres` (from
`select/linear`).
* Allow `SlurmUser`/`root` to use reservations without specific
permissions.
* Fix sending step signals to nodes not allocated by the step.
* Remove `CgroupAutomount=` option from `cgroup.conf`.
* Add `TopologyRoute=RoutePart` to route communications based
on partition node lists.
* Added ability for configless to push Prolog and Epilog
scripts to `slurmd`s.
* Prolog and Epilog do not have to be fully qualified pathnames.
* Changed default value of `PriorityType` from `priority/basic`
to `priority/multifactor`.
* `torque/mpiexec` - Propogate exit code from `launched` process.
* `slurmrestd` - Add new rlimits fields for job submission.
* Define SPANK options environment variables when
`--export=[NIL|NONE]` is specified.
* `slurmrestd` - Numeric input fields provided with a null
formatted value will now convert to zero (0) where it can be
a valid value. This is expected to be only be notable with job
submission against v0.0.38 versioned endpoints with job
requests with fields provided with null values. These fields
were already rejected by v0.0.39+ endpoints, unless `+complex`
parser value is provided to v0.0.40+ endpoints.
* `slurmrestd` - Improve parsing of integers and floating point
numbers when handling incoming user provided numeric fields.
Fields that would have not rejected a number for a numeric
field followed by other non-numeric characters will now get
rejected. This is expected to be only be notable with job
submission against v0.0.38 versioned endpoints with malformed
job requests.
* Reject reservation update if it will result in previously
submitted jobs losing access to the reservation.
* `data_parser/v0.0.40` - output partition state when dumping
partitions.
* Allow for a shared suffix to be used with the hostlist format.
E.g., `node[0001-0010]-int`.
* Replace `SRUN_CPUS_PER_TASK` with `SLURM_CPUS_PER_TASK` and
get back the previous behavior before Slurm 22.05 since now we
have the new external launcher step.
* `job_container/tmpfs` - Add `BasePath=none` option to disable
plugin on node subsets when there is a global setting.
* Add QOS flag `Relative`. If set the QOS limits will be treated
as percentages of a cluster/partition instead of absolutes.
* Remove `FIRST_CORES` flag from reservations.
* Add cloud instance id and instance type to node records.
Can be viewed/updated with `scontrol`.
* `slurmd` - add `instance-id`, `instance-type`, and `extra`
options to allow them to be set on startup.
* Add cloud instance accounting to database that can be viewed
with `sacctmgr show instance`.
* `select/linear` - fix task launch failure that sometimes
occurred when requesting `*threads-per-core` or
`--hint=nomultithread`. This also fixes memory calculation
with one of these options and `*mem-per-cpu`:
Previously, memory = mem-per-cpu * all cpus including unusable
threads.
Now, memory = mem-per-cpu * only usuable threads. This
behavior matches the documentation and select/cons_tres.
* `gpu/nvml` - Reduce chances of `NVML_ERROR_INSUFFICIENT_SIZE`
error when getting gpu memory information.
* `slurmrestd` - Convert to generating `OperationIDs` based on
path for all v0.0.40 tagged paths.
* `slurmrestd` - Reduce memory used while dumping a job's stdio
paths.
* `slurmrestd` - Jobs queried from `data_parser/v0.0.40` from
`slurmdb` will have `step/id` field given as a string to match
CLI formatting instead of an object.
* `sacct` - Output in JSON or YAML output will will have the
`step/id` field given as a string instead of an object.
* `scontrol`/`squeue` - Step output in JSON or YAML output will
will have the `id` field given as a string instead of an
object.
* `slurmrestd` - For `GET /slurmdb/v0.0.40/jobs` mimick default
behavior for handling of job start and end times as `sacct`
when one or both fields are not provided as a query parameter.
* `openapi/slurmctld` - Add `GET /slurm/v0.0.40/shares` endpoint
to dump same output as `sshare`.
* `sshare` - add JSON/YAML support.
* `data_parser/v0.0.40` - Remove `required/memory` output in
json. It is replaced by `required/memory_per_cpu` and
`required/memory_per_node`.
* `slurmrestd` - Add numeric id to all association identifiers
to allow unique identification where association has been
deleted but is still referenced by accounting record.
* `slurmrestd` - Add accounting, id, and comment fields to
association dumps.
* Use `memory.current` in cgroup/v2 instead of manually
calculating RSS. This makes accounting consistent with
OOM Killer.
* `sreport` - cluster Utilization `PlannedDown` field now
includes the time that all nodes were in the `POWERED_DOWN`
state instead of just cloud nodes.
* `scontrol` update partition now allows `Nodes+=<node-list>` and
`Nodes-=<node-list>` to add/delete nodes from the existing
partition node list. `Nodes=+host1,-host2` is also allowed.
* `sacctmgr` - add `--yaml` and `--json` arg support to specify
`data_parser` plugin.
* `sacctmgr` can now modify QOS's RawUsage to zero or a positive
value.
* `sdiag` - Added statistics on why the main and backfill
schedulers have stopped evaluation on each scheduling cycle.
the number of `RPC limit exceeded...` messages that are logged.
* Rename `sbcast --fanout` to `--treewidth`.
* Remove `SLURM_NODE_ALIASES` env variable.
* Enable fanout for dynamic and unaddresable cloud nodes.
* Fix how steps are dealloced in an allocation if the last step
of an srun never completes due to a node failure.
* Remove redundant database indexes.
* Add database index to suspend table to speed up archive/purges.
* When requesting `--tres-per-task` alter incorrect request for
TRES, it should be `TRESType/TRESName` not `TRESType:TRESName`.
* Make it so reservations can reserve GRES.
* `sbcast` - use the specified `--fanout` value on all hops in
message forwarding; previously the specified fanout was only
used on the first hop, and additional hops used `TreeWidth` in
`slurm.conf`.
* `slurmrestd`- remove logger prefix from `-s/-a list` options
outputs.
* `switch/hpe_slingshot` - Add support for collectives.
* Nodes with suspended jobs can now be displayed as `MIXED`.
* Fix inconsistent handling of using cli and/or environment
options for `tres_per_task=cpu:#` and `cpus_per_gpu`.
* Requesting `--cpus-per-task` will now set
`SLURM_TRES_PER_TASK=cpu:#` in the environment.
* For some tres related environment variables such as
`SLURM_TRES_PER_TASK`, when `srun` requests a different value
for that option, set these environment variables to the value
requested by `srun`. Previously these environment variables
were unchanged from the job allocation. This bug only affected
the output environment variables, not the actual step resource
allocation.
* `RoutePlugin=route/topology` has been replaced with
`TopologyParam=RouteTree`.
* If `ThreadsPerCore` in `slurm.conf` is configured with less
than the number of hardware threads, fix a bug where the task
plugins used fewer cores instead of using fewer threads per core.
* Fix arbitrary distribution allowing it to be used with `salloc`
and `sbatch` and fix how cpus are allocated to nodes.
* Allow nodes to reboot while node is drained or in a
maintenance state.
* Allow `scontrol` reboot to use nodesets to filter nodes to reboot.
* Fix how the topology of typed gres gets updated.
* Changes to the Type option in gres.conf now can be applied with
`scontrol` reconfig.
* Allow for jobs that request a newly configured gres type to be
queued even when the needed `slurmd`s have not yet registered.
* Kill recovered jobs that require unconfigured gres types.
* If keepalives are configured, enable them on all persistent
connections.
* Configless - Also send Includes from configuration files not
parsed by the controller (i.e. from `plugstack.conf`).
* Add `gpu/nrt` plugin for nodes using Trainium/Inferentia
devices.
* `data_parser/v0.0.40` - Add `START_RECEIVED` to job flags in
dumped output.
* SPANK - Failures from most spank functions (not epilog or
exit) will now cause the step to be marked as failed and the
command (`srun`, `salloc`, `sbatch *wait`) to return 1.
* Wed Jan 03 2024 Egbert Eich <eich@suse.com>
- Update to 23.02.6 to fix (CVE-2023-49933 - bsc#1218046, CVE-2023-49935 -
bsc#1218049, CVE-2023-49936 - bsc#1218050, CVE-2023-49937 - bsc#1218051,
CVE-2023-49938 - bsc#1218053)
* Security Fixes:
+ Add `JobAcctGatherParams=DisableGPUAcct` to disable gpu accounting.
+ `acct_gather_energy/ipmi` - Improve logging of DCMI issues.
+ `gpu/oneapi` - Add support for new env vars `ZE_FLAT_DEVICE_HIERARCHY`
and `ZE_ENABLE_PCI_ID_DEVICE_ORDER`.
+ `data_parser/v0.0.39` - skip empty string when parsing QOS ids.
+ Remove error message from `assoc_mgr_update_assocs` when purposefully
resetting the default QOS.
* Bug Fixes:
+ `libslurm_nss` - Avoid causing glibc to assert due to an unexpected
return from slurm_nss due to an error during lookup.
+ Fix job requests with `--tres-per-task` sometimes resulting in bad
allocations that cannot run subsequent job steps.
+ Fix issue with `slurmd` where `srun` fails to be warned when a node
prolog script runs beyond `MsgTimeout` set in `slurm.conf`.
+ `gres/shard` - Fix plugin functions to have matching parameter orders.
+ `gpu/nvml` - Fix issue that resulted in the wrong MIG devices being
constrained to a job
+ `gpu/nvml` - Fix linking issue with MIGs that prevented multiple MIGs
being used in a single job for certain MIG configurations
+ Fix file descriptor leak in slurmd when using `acct_gather_energy/ipmi`
with DCMI devices.
+ `sview` - avoid crash when job has a node list string > 49 characters.
+ Prevent `slurmctld` crash during reconfigure when packing job start
messages.
+ Preserve reason uid on reconfig.
+ Update node reason with updated `INVAL` state reason if different from
last registration.
+ `conmgr` - Avoid NULL dereference when using `auth/none`.
+ `data_parser/v0.0.39` - Fixed how deleted QOS and associations for jobs
are dumped.
+ `burst_buffer/lua` - fix stage in counter not decrementing when a job is
cancelled during stage in. This counter is used to enforce the limit of
128 scripts per stage.
+ `data_parser/v0.0.39` - Fix how the `INVALID` nodes state is dumped.
+ `data_parser/v0.0.39` - Fix parsing of flag arrays to allow muliple
flags to be set.
+ Avoid leaking sockets when an x11 application is closed in an allocation.
+ Fix missing mutex unlock in group cache code which could cause slurmctld
to freeze.
+ Fix scrontab monthly jobs possibly skipping a month if added near the
end of the month.
+ Fix loading of the gpu account gather energy plugin.
+ Fix `slurmctld` segfault when reconfiguring after a job resize.
+ Fix crash in slurmstepd that can occur when launching tasks via mpi using
the `pmi2` plugin and using the `route/topology` plugin.
+ Fix `qos <id> doesn't exist` error message in `assoc_mgr_update_assocs`
to print the attempted new default qos, rather than the current default
qos.
+ `data_parser/v0.0.39` - Fix segfault when POSTing data with association
usage.
* Other Changes and Improvements:
+ Prevent message extension attacks that could bypass the message hash.
CVE-2023-49933.
+ Prevent message hash bypass in slurmd which can allow an attacker to
reuse root-level MUNGE tokens and escalate permissions. CVE-2023-49935.
+ Prevent NULL pointer dereference on `size_valp` overflow. CVE-2023-49936.
+ Prevent double-xfree() on error in `_unpack_node_reg_resp()`.
CVE-2023-49937.
+ Prevent modified `sbcast` RPCs from opening a file with the wrong group
permissions. CVE-2023-49938.
- Fix %do_obsoletes macro expansion to work with SLE-12.
* Thu Nov 30 2023 Egbert Eich <eich@suse.com>
- Add missing service file for slurmrestd (boo#1217711).
* Tue Nov 28 2023 Egbert Eich <eich@suse.com>
- Explicitly create an Obsoletes: entry for each package version
that is obsoleted by the present version. These are all published
versions of the last two major releases as well as all minor
versions of the present release lower than the current one
(bsc#1216869 2nd part).
This prevents the current version to upgrade a old Slurm version
for which no upgrade path exists.
* Mon Nov 20 2023 Egbert Eich <eich@suse.com>
- On SLE-12 exclude build for s390x.
* Mon Nov 06 2023 Egbert Eich <eich@suse.com>
- Add missing dependencies to slurm-config to plugins package.
These should help to tie down the slurm version and help to avoid
a package mix (bsc#1216869).
* Thu Oct 12 2023 Christian Goll <cgoll@suse.com>
- update to 23.02.6 to fix (CVE-2023-41914, bsc#1216207)
* Removed Fix-test-32.8.patch as fixed upstream
* Bug Fixes:
+ Fix `CpusPerTres=` not upgreadable with scontrol update
+ Fix unintentional gres removal when validating the gres job state.
+ Fix `--without-hpe-slingshot` configure option.
+ Fix cgroup v2 memory calculations when transparent huge pages are used.
+ Fix parsing of `sgather --timeout` option.
+ Fix regression from 22.05.0 that caused `srun --cpu-bind "=verbose"`
and `"=v"` options give different CPU bind masks.
+ Fix "_find_node_record: lookup failure for node" error message appearing
for all dynamic nodes during reconfigure.
+ Avoid segfault if loading serializer plugin fails.
+ `slurmrestd` - Correct OpenAPI format for `GET /slurm/v0.0.39/licenses`.
+ `slurmrestd` - Correct OpenAPI format for
`GET /slurm/v0.0.39/job/{job_id}`.
+ `slurmrestd` - Change format to multiple fields in
`GET /slurmdb/v0.0.39/assocations` and `GET /slurmdb/v0.0.39/qos` to
handle infinite and unset states.
+ When a node fails in a job with `--no-kill`, preserve the extern step on the
remaining nodes to avoid breaking features that rely on the extern step
such as `pam_slurm_adopt`, `x11`, and `job_container/tmpfs`.
+ `auth/jwt` - Ignore `x5c` field in JWKS files.
+ `auth/jwt` - Treat 'alg' field as optional in JWKS files.
+ Allow job_desc.selinux_context to be read from the job_submit.lua script.
+ Skip check in slurmstepd that causes a large number of errors in the
munge log: "Unauthorized credential for client UID=0 GID=0".
This error will still appear on `slurmd`/`slurmctld`/`slurmdbd` start up
and is not a cause for concern.
+ `slurmctld` - Allow startup with zero partitions.
+ Fix some mig profile names in slurm not matching nvidia mig profiles.
+ Prevent `slurmscriptd` processing delays from blocking other threads in
`slurmctld` while trying to launch `{Prolog|Epilog}Slurmctld`.
+ Fix sacct printing ReqMem field when memory doesn't exist in requested
TRES.
+ Fix how heterogenous steps in an allocation with `CR_PACK_NODE` or
`-mpack` are created.
+ Fix `slurmctld` crash from race condition within `job_submit_throttle`
plugin.
+ Fix `--with-systemdsystemunitdir` when requesting a default location.
+ Fix not being able to cancel an array task by the jobid (i.e. not
`<jobid>_<taskid>`) through scancel, job launch failure or prolog
failure.
+ Fix cancelling the whole array job when the array task is the meta job
and it fails job or prolog launch and is not requeable. Cancel only the
specific task instead.
+ Fix regression in 21.08.2 where MailProg did not run for `mail-type=end`
for jobs with non+zero exit codes.
+ Fix incorrect setting of memory.swap.max in cgroup/v2.
+ Fix `jobacctgather/cgroup` collection of disk/io, gpumem, gpuutil TRES
values.
+ Fix -d singleton for heterogeneous jobs.
+ Downgrade info logs about a job meeting a "maximum node limit" in the
select plugin to `DebugFlags=SelectType`. These info logs could spam the
slurmctld log file under certain circumstances.
+ `prep/script` - Fix `[Srun|Task]<Prolog|Epilog>` missing
`SLURM_JOB_NODELIST`.
+ gres - Rebuild GRES core bitmap for nodes at startup. This fixes error:
"Core bitmaps size mismatch on node [HOSTNAME]", which causes jobs to
enter state "Requested node configuration is not available".
+ `slurmctd` - Allow startup with zero nodes.
+ Fix filesystem handling race conditions that could lead to an attacker
taking control of an arbitrary file, or removing entire directories'
contents. CVE-2023-41914.
* Mon Sep 18 2023 Egbert Eich <eich@suse.com>
- Updated to version 23.02.5 with the following changes:
* Bug Fixes:
+ Revert a change in 23.02 where `SLURM_NTASKS` was no longer set in the
job's environment when `--ntasks-per-node` was requested.
The method that is is being set, however, is different and should be more
accurate in more situations.
+ Change pmi2 plugin to honor the `SrunPortRange` option. This matches the
new behavior of the pmix plugin in 23.02.0. Note that neither of these
plugins makes use of the `MpiParams=ports=` option, and previously
were only limited by the systems ephemeral port range.
+ Fix regression in 23.02.2 that caused slurmctld -R to crash on startup if
a node features plugin is configured.
+ Fix and prevent reoccurring reservations from overlapping.
+ `job_container/tmpfs` - Avoid attempts to share BasePath between nodes.
+ With `CR_Cpu_Memory`, fix node selection for jobs that request gres and
`--mem-per-cpu`.
+ Fix a regression from 22.05.7 in which some jobs were allocated too few
nodes, thus overcommitting cpus to some tasks.
+ Fix a job being stuck in the completing state if the job ends while the
primary controller is down or unresponsive and the backup controller has
not yet taken over.
+ Fix `slurmctld` segfault when a node registers with a configured
`CpuSpecList` while `slurmctld` configuration has the node without
`CpuSpecList`.
+ Fix cloud nodes getting stuck in `POWERED_DOWN+NO_RESPOND` state after
not registering by `ResumeTimeout`.
+ `slurmstepd` - Avoid cleanup of `config.json-less` containers spooldir
getting skipped.
+ Fix scontrol segfault when 'completing' command requested repeatedly in
interactive mode.
+ Properly handle a race condition between `bind()` and `listen()` calls
in the network stack when running with SrunPortRange set.
+ Federation - Fix revoked jobs being returned regardless of the
`-a`/`--all` option for privileged users.
+ Federation - Fix canceling pending federated jobs from non-origin
clusters which could leave federated jobs orphaned from the origin
cluster.
+ Fix sinfo segfault when printing multiple clusters with `--noheader`
option.
+ Federation - fix clusters not syncing if clusters are added to a
federation before they have registered with the dbd.
+ `node_features/helpers` - Fix node selection for jobs requesting
changeable.
features with the `|` operator, which could prevent jobs from
running on some valid nodes.
+ `node_features/helpers` - Fix inconsistent handling of `&` and `|`,
where an AND'd feature was sometimes AND'd to all sets of features
instead of just the current set. E.g. `foo|bar&baz` was interpreted
as `{foo,baz}` or `{bar,baz}` instead of how it is documented:
`{foo} or {bar,baz}`.
+ Fix job accounting so that when a job is requeued its allocated node
count is cleared. After the requeue, sacct will correctly show that
the job has 0 `AllocNodes` while it is pending or if it is canceled
before restarting.
+ `sacct` - `AllocCPUS` now correctly shows 0 if a job has not yet
received an allocation or if the job was canceled before getting one.
+ Fix intel OneAPI autodetect: detect the `/dev/dri/renderD[0-9]+` GPUs,
and do not detect `/dev/dri/card[0-9]+`.
+ Fix node selection for jobs that request `--gpus` and a number of
tasks fewer than GPUs, which resulted in incorrectly rejecting these
jobs.
+ Remove `MYSQL_OPT_RECONNECT` completely.
+ Fix cloud nodes in `POWERING_UP` state disappearing (getting set
to `FUTURE`)
when an `scontrol reconfigure` happens.
+ `openapi/dbv0.0.39` - Avoid assert / segfault on missing coordinators
list.
+ `slurmrestd` - Correct memory leak while parsing OpenAPI specification
templates with server overrides.
+ Fix overwriting user node reason with system message.
+ Prevent deadlock when `rpc_queue` is enabled.
+ `slurmrestd` - Correct OpenAPI specification generation bug where
fields with overlapping parent paths would not get generated.
+ Fix memory leak as a result of a partition info query.
+ Fix memory leak as a result of a job info query.
+ For step allocations, fix `--gres=none` sometimes not ignoring gres
from the job.
+ Fix `--exclusive` jobs incorrectly gang-scheduling where they shouldn't.
+ Fix allocations with `CR_SOCKET`, gres not assigned to a specific
socket, and block core distribion potentially allocating more sockets
than required.
+ Revert a change in 23.02.3 where Slurm would kill a script's process
group as soon as the script ended instead of waiting as long as any
process in that process group held the stdout/stderr file descriptors
open. That change broke some scripts that relied on the previous
behavior. Setting time limits for scripts (such as
`PrologEpilogTimeout`) is strongly encouraged to avoid Slurm waiting
indefinitely for scripts to finish.
+ Fix `slurmdbd -R` not returning an error under certain conditions.
+ `slurmdbd` - Avoid potential NULL pointer dereference in the mysql
plugin.
+ Fix regression in 23.02.3 which broken X11 forwarding for hosts when
MUNGE sends a localhost address in the encode host field. This is caused
when the node hostname is mapped to 127.0.0.1 (or similar) in
`/etc/hosts`.
+ `openapi/[db]v0.0.39` - fix memory leak on parsing error.
+ `data_parser/v0.0.39` - fix updating qos for associations.
+ `openapi/dbv0.0.39` - fix updating values for associations with null
users.
+ Fix minor memory leak with `--tres-per-task` and licenses.
+ Fix cyclic socket cpu distribution for tasks in a step where
`--cpus-per-task` < usable threads per core.
+ `slurmrestd` - For `GET /slurm/v0.0.39/node[s]`, change format of
node's energy field `current_watts` to a dictionary to account for
unset value instead of dumping 4294967294.
+ `slurmrestd` - For `GET /slurm/v0.0.39/qos`, change format of QOS's
field "priority" to a dictionary to account for unset value instead of
dumping 4294967294.
+ slurmrestd - For `GET /slurm/v0.0.39/job[s]`, the 'return code'
code field in `v0.0.39_job_exit`_code will be set to -127 instead of
being left unset where job does not have a relevant return code.
* Other Changes:
+ Remove --uid / --gid options from salloc and srun commands. These options
did not work correctly since the CVE-2022-29500 fix in combination with
some changes made in 23.02.0.
+ Add the `JobId` to `debug()` messages indicating when
`cpus_per_task/mem_per_cpu` or `pn_min_cpus` are being automatically
adjusted.
+ Change the log message warning for rate limited users from verbose to
info.
+ `slurmstepd` - Cleanup per task generated environment for containers in
spooldir.
+ Format batch, extern, interactive, and pending step ids into strings that
are human readable.
+ `slurmrestd` - Reduce memory usage when printing out job CPU frequency.
+ `data_parser/v0.0.39` - Add `required/memory_per_cpu` and
`required/memory_per_node` to `sacct --json` and `sacct --yaml` and
`GET /slurmdb/v0.0.39/jobs` from slurmrestd.
+ `gpu/oneapi` - Store cores correctly so CPU affinity is tracked.
+ Allow `slurmdbd -R` to work if the root assoc id is not 1.
+ Limit periodic node registrations to 50 instead of the full `TreeWidth`.
Since unresolvable `cloud/dynamic` nodes must disable fanout by setting
`TreeWidth` to a large number, this would cause all nodes to register at
once.
* Mon Aug 21 2023 Christian Goll <cgoll@suse.com>
- Updated to 23.02.4 with the following changes:
* Bug Fixes:
+ Fix main scheduler loop not starting after a failover to backup
controller. Avoid slurmctld segfault when specifying
`AccountingStorageExternalHost` (bsc#1214983).
+ Fix sbatch return code when `--wait` is requested on a job array.
+ Fix collected `GPUUtilization` values for `acct_gather_profile` plugins.
+ Fix `slurmrestd` handling of job hold/release operations.
+ Fix step running indefinitely when slurmctld takes more than
`MessageTimeout` to respond. Now, `slurmctld` will cancel the step when
detected, preventing following steps from getting stuck waiting for
resources to be released.
+ Fix regression to make `job_desc.min_cpus` accurate again in `job_submit`
when requesting a job with `--ntasks-per-node`.
+ Fix handling of `ArrayTaskThrottle` in backfill.
+ Fix regression in 23.02.2 when checking gres state on `slurmctld`
startup or reconfigure. Gres changes in the configuration were not
updated on slurmctld startup. On startup or reconfigure, these messages
were present in the log: `error: Attempt to change gres/gpu Count`.
+ Fix potential double count of gres when dealing with limits.
+ Fix `slurmstepd` segfault when `ContainerPath` is not set in `oci.conf`
+ Fixed an issue where jobs requesting licenses were incorrectly rejected.
+ `scrontab` - Fix cutting off the final character of quoted variables.
+ `smail` - Fix issues where e-mails at job completion were not being sent.
+ `scontrol/slurmctld` - fix comma parsing when updating a reservation's
nodes.
+ Fix `--gpu-bind=single binding` tasks to wrong gpus, leading to some gpus
having more tasks than they should and other gpus being unused.
+ Fix regression in 23.02 that causes slurmstepd to crash when `srun`
requests more than `TreeWidth` nodes in a step and uses the pmi2 or
pmix plugin.
+ `job_container/tmpfs` - Fix `%h` and `%n` substitution in `BasePath`
where `%h` was substituted as the NodeName instead of the hostname,
and %n was substituted as an empty string.
+ Fix regression where `--cpu-bind=verbose` would override
`TaskPluginParam`.
+ `scancel` - Fix `--clusters/-M` for federations. Only filtered jobs
(e.g. `-A`, `-u`, `-p`, etc.) from the specified clusters will be
canceled, rather than all jobs in the federation. Specific jobids
will still be routed to the origin cluster for cancellation.
* Other changes:
+ Make spank `S_JOB_ARGV` item value hold the requested command `argv`
instead of the `srun --bcast` value when `--bcast` requested (only in
local context).
+ `scontrol` - Permit changes to StdErr and StdIn for pending jobs.
+ `scontrol` - Reset `std`{`err`,`in`,`out`} when set to empty string.
+ `slurmrestd` - mark environment as a required field for job submission
descriptions.
+ `slurmrestd` - avoid dumping null in OpenAPI schema required fields.
+ `data_parser/v0.0.39` - avoid rejecting valid `memory_per_node` formatted
as dictionary provided with a job description.
+ `data_parser/v0.0.39` - avoid rejecting valid `memory_per_cpu` formatted
as dictionary provided with a job description.
+ `slurmrestd` - Return HTTP error code 404 when job query fails.
+ `slurmrestd` - Add return schema to error response to job and license
query.
+ Change the log message warning for rate limited users from debug to
verbose.
+ `cgroup/v2` - Avoid capturing log output for ebpf when constraining
devices,
as this can lead to inadvertent failure if the log buffer is too small.
+ Added error message when attempting to use sattach on batch or extern
steps.
+ Reject job `ArrayTaskThrottle` update requests from unprivileged users.
+ `data_parser/v0.0.39` - populate description fields of property objects
in generated OpenAPI specifications where defined.
+ `slurmstepd` - Avoid segfault caused by `ContainerPath` not being
terminated by `/` in `oci.conf`.
+ `data_parser/v0.0.39` - Change `v0.0.39_job_info` response to tag
`exit_code` field as being complex instead of only an unsigned integer.
- Updated to 23.02.3 with the following changes:
* Bug Fixes:
+ `slurmctld` - Fix backup slurmctld crash when it takes control
multiple times.
+ Fix regression in 23.02.2 that ignored the partition `DefCpuPerGPU`
setting on the first pass of scheduling a job requesting
`--gpus --ntasks`.
+ `srun` - fix issue creating regular and interactive steps because
environment variables were incorrectly set on non-HetSteps.
+ Fix dynamic nodes getting stuck in allocated states when reconfiguring.
+ Fix regression in 23.02.2 that set the `SLURM_NTASKS` environment
variable in sbatch jobs from `--ntasks-per-node` when `--ntasks` was not
requested.
+ Fix regression in 23.02 that caused sbatch jobs to set the wrong number
of tasks when requesting `--ntasks-per-node` without `--ntasks`, and also
requesting one of the following options: `--sockets-per-node`,
`--cores-per-socket`, `--threads-per-core` (or `--hint=nomultithread`),
or `-B,--extra-node-info`.
+ Fix double counting suspended job counts on nodes when reconfiguring,
which prevented nodes with suspended jobs from being powered down or
rebooted once the jobs completed.
+ Fix backfill not scheduling jobs submitted with `--prefer` and
`--constraint` properly.
+ mpi/pmix - fix regression introduced in 23.02.2 which caused PMIx shmem
backed files permissions to be incorrect.
+ api/submit - fix memory leaks when submission of batch regular jobs
or batch HetJobs fails (response data is a return code).
+ Fix regression in 23.02 leading to error() messages being sent at `INFO`
instead of `ERR` in syslog.
+ Fix `TresUsageIn[Tot|Ave]` calculation for `gres/gpumem` and
`gres/gpuutil`.
+ Fix issue in the gpu plugins where gpu frequencies would only be set if
both gpu memory and gpu frequencies were set, while one or the other
suffices.
+ Fix reservations group ACL's not working with the root group.
+ Fix updating a job with a ReqNodeList greater than the job's node count.
+ Fix inadvertent permission denied error for `--task-prolog` and
`--task-epilog` with filesystems mounted with `root_squash`.
+ Fix missing detailed cpu and gres information in json/yaml output from
`scontrol`, `squeue` and `sinfo`.
+ Fix regression in 23.02 that causes a failure to allocate job steps that
request `--cpus-per-gpu` and gpus with types.
+ Fix potentially waiting indefinitely for a defunct process to finish,
which affects various scripts including `Prolog` and `Epilog`. This could
have various symptoms, such as jobs getting stuck in a completing state.
+ Fix losing list of reservations on job when updating job with list of
reservations and restarting the controller.
+ Fix nodes resuming after down and drain state update requests from
clients older than 23.02.
+ Fix advanced reservation creation/update when an association that should
have access to it is composed with partition(s).
+ Fix job layout calculations with `--ntasks-per-gpu`, especially when
`--nodes` has not been explicitly provided.
+ Fix X11 forwarding for jobs submitted from the slurmctld host.
+ When a job requests `--no-kill` and one or more nodes fail during the
job, fix subsequent job steps unable to use some of the remaining
resources allocated to the job.
+ Fix shared gres allocation when using `--tres-per-task` with tasks that
span multiple sockets.
+ `auth/jwt` - Fix memory leak.
* Other changes:
+ `openapi/dbv0.0.39/users` - If a default account update failed, resulting
in a no-op, the query returned success without any warning. Now a warning
is sent back to the client that the default account wasn't modified.
+ Avoid job write lock when nodes are dynamically added/removed.
+ `burst_buffer/lua` - allow jobs to get scheduled sooner after
`slurm_bb_data_in` completes.
+ `openapi/v0.0.39` - fix memory leak in `_job_post_het_submit()`.
+ Avoid possible `slurmctld` segfault caused by race condition with already
completed `slurmdbd_conn` connections.
+ `Slurmdbd.conf` checks included conf files for 0600 permissions
+ `slurmrestd` - fix regression "oversubscribe" fields were removed from
job descriptions and submissions from v0.0.39 end points.
+ `accounting_storage/mysql` - Query for indiviual QOS correctly when you
have more than 10.
+ Add warning message about ignoring `--tres-per-tasks=license` when used
on a step.
+ `sshare` - Fix command to work when using `priority/basic`.
+ Avoid loading `cli_filter` plugins outside of `salloc`/`sbatch`/`scron`/
`srun`. This fixes a number of missing symbol problems that can manifest
for executables linked against libslurm (and not `libslurmfull`).
+ Allow cloud_reg_addrs to update dynamically registered node's addrs on
subsequent registrations.
+ Revert a change in 22.05.5 that prevented tasks from sharing a core if
`--cpus-per-task` > threads per core, but caused incorrect accounting and
cpu binding. Instead, `--ntasks-per-core=1` may be requested to prevent
tasks from sharing a core.
+ Correctly send `assoc_mgr` lock to mcs plugin.
+ Avoid unnecessary `gres/gpumem` and `gres/gpuutil` `TRES` position
lookups.
+ `sacct` - when printing `PLANNED` time, use end time instead of start
time for jobs cancelled before they started.
+ Hold the job with `(Reservation ... invalid)` state reason if the
reservation is not usable by the job.
+ `sbatch` - Added new `--export=NIL` option.
- Removed:
* Fix-test-3.13.patch
* Fix-test-38.11.patch as both tests changed upstream
* Tue Jul 04 2023 Egbert Eich <eich@suse.com>
- Create a macro for upgrade dependency to ensure uniform handling.
* Tue May 09 2023 Christian Goll <cgoll@suse.com>
- updated to 23.02.02 which includes a number of fixes to Slurm stability
* Includes a fix for a regression in 23.02 that caused openmpi mpirun to fail
to launch tasks.
* It also includes two functional changes: Don't update the cron job tasks if
the whole crontab file is left untouched after opening it with scrontab -e
* Sort dynamic nodes and include them in topology after scontrol reconfigure
or a slurmctld restart.
* Mon Apr 17 2023 Egbert Eich <eich@suse.com>
- Web-configurator: changed presets to SUSE defaults.
- If %_restart_on_update is no longer defined replace by own
macro.
- Marked slurm-openlava, slurm-seff and slurm-sjstat noarch.
- rpmlint:
* dropped some rpmlint filters which are no longer relevant.
* added/refreshed filters. For Details, see rpmlintrc.
- Remove workaround to fix the restart issue in an Slurm package
described in bsc#1088693.
The Slurm version in this package as 16.05. Any attempt to
directly migrate to the current version is bound to fail
anyway.
- Now require slurm-munge if munge authentication is installed.
* Fri Mar 31 2023 Christian Goll <cgoll@suse.com>
- updated to 23.02.1 with the following changes:
* job_container/tmpfs - cleanup job container even if namespace mount is
already unmounted.
* openapi/dbv0.0.38 - Fix not displaying an error when updating QOS or
associations fails.
* Fix nodes remaining as PLANNED after slurmctld save state recovery.
* Add cgroup.conf EnableControllers option for cgroup/v2.
* Get correct cgroup root to allow slurmd to run in containers like Docker.
* slurmctld - add missing PrivateData=jobs check to step ContainerID lookup
requests originated from 'scontrol show step container-id=<id>' or certain
scrun operations when container state can't be directly queried.
* Fix nodes un-draining after being drained due to unkillable step.
* Fix remote licenses allowed percentages reset to 0 during upgrade.
* sacct - Avoid truncating time strings when using SLURM_TIME_FORMAT with
the --parsable option.
* Fix regression in 22.05.0rc1 that broke Nodes=ALL in a NodeSet.
* openapi/v0.0.39 - fix jobs submitted via slurmrestd being allocated fewer
CPUs than tasks when requesting multiple tasks.
* Fix job not being scheduled on valid nodes and potentially being rejected
when using parentheses at the beginning of square brackets in a feature
request, for example: "feat1&[(feat2|feat3)]".
* Fix regression in 23.02.0rc1 which made --gres-flags=enforce-binding no
longer enforce optimal core-gpu job placement.
* mpi/pmix - Fix v5 to load correctly when libpmix.so isn't in the normal
lib path.
* data_parser/v0.0.39 - fix regression where "memory_per_node" would be
rejected for job submission.
* data_parser/v0.0.39 - fix regression where "memory_per_cpu" would be
rejected for job submission.
* slurmctld - add an assert to check for magic number presence before deleting
a partition record and clear the magic afterwards to better diagnose
potential memory problems.
* Clean up OCI containers task directories correctly.
* scrun - Run under existing job when SLURM_JOB_ID is present.
* Prevent a slurmstepd crash when the I/O subsystem has hung.
* common/conmgr - fix memory leak of complete connection list.
* job_container/tmpfs - avoid printing extraneous error messages when running
a spank plugin that implements slurm_spank_job_prolog() or
slurm_spank_job_epilog().
* Fix srun < 23.02 always getting an "exact" core allocation.
* Prevent scontrol < 23.02 from setting MaxCPUsPerSocket to 0.
* Add ScronParameters=explicit_scancel and corresponding scancel --cron
option.
- removed right-pmix-path.patch as fixed upstream
* Thu Mar 16 2023 Christian Goll <cgoll@suse.com>
- use libpmix.so.2 instead of libpmix.so to fix (bsc#1209260)
this removes the need of pmix-pluginlib
added: right-pmix-path.patch
* Wed Mar 15 2023 Christian Goll <cgoll@suse.com>
- slurm-plugins need to require pmix-pluginlib (bsc#1209260)
* Tue Mar 07 2023 Egbert Eich <eich@suse.com>
- Stop pulling firewall rules from github. There is no benefit to
host these separately.
- Remove pre-sle12 pieces.
* Wed Mar 01 2023 Egbert Eich <eich@suse.com>
- Add missing Provides:, Conflicts: and Obsoletes: to slurm-cray,
slurm-hdf5 and slurm-testsuite to avoid package conflicts.
- Unify Obsoletes:.
- Consolidate spec files between different Slurm releases in
Leap/SLE maintenance.
- Add dependency for the general plugin package to the
AcctGatherProfile HDF5 plugin.
- Adjust node RealMemory in slurm.conf of test suite for 8G test
nodes.
* Mon Feb 20 2023 Egbert Eich <eich@suse.com>
- updated to 23.02.0
* Highlights
+ slurmctld - Add new RPC rate limiting feature. This is enabled through
SlurmctldParameters=rl_enable, otherwise disabled by default.
+ Make scontrol reconfigure and sending a SIGHUP to the slurmctld behave
the same. If you were using SIGHUP as a 'lighter' scontrol reconfigure
to rotate logs please update your scripts to use SIGUSR2 instead.
+ Change cloud nodes to show by default. PrivateData=cloud is no longer
needed.
+ sreport - Count planned (FKA reserved) time for jobs running in
IGNORE_JOBS reservations. Previously was lumped into IDLE time.
+ job_container/tmpfs - Support running with an arbitrary list of private
mount points (/tmp and /dev/shm are the default, but not required).
+ job_container/tmpfs - Set more environment variables in InitScript.
+ Make all cgroup directories created by Slurm owned by root. This was the
behavior in cgroup/v2 but not in cgroup/v1 where by default the step
directories ownership were set to the user and group of the job.
+ accounting_storage/mysql - change purge/archive to calculate record ages
based on end time, rather than start or submission times.
+ job_submit/lua - add support for log_user() from slurm_job_modify().
+ Run the following scripts in slurmscriptd instead of slurmctld:
ResumeProgram, ResumeFailProgram, SuspendProgram, ResvProlog, ResvEpilog,
and RebootProgram (only with SlurmctldParameters=reboot_from_controller).
+ Only permit changing log levels with 'srun --slurmd-debug' by root
or SlurmUser.
+ slurmctld will fatal() when reconfiguring the job_submit plugin fails.
+ Add PowerDownOnIdle partition option to power down nodes after nodes
become idle.
+ Add "[jobid.stepid]" prefix from slurmstepd and "slurmscriptd" prefix
from slurmcriptd to Syslog logging. Previously was only happening when
logging to a file.
+ Add purge and archive functionality for job environment and job batch
script records.
+ Extend support for Include files to all "configless" client commands.
+ Make node weight usable for powered down and rebooting nodes.
+ Removed 'launch' plugin.
+ Add "Extra" field to job to store extra information other than a comment.
+ Add usage gathering for AMD (requires ROCM 5.5+) and NVIDIA gpus.
+ Add job's allocated nodes, features, oversubscribe, partition, and
reservation to SLURM_RESUME_FILE output for power saving.
+ Automatically create directories for stdout/stderr output files. Paths
may use %j and related substitution characters as well.
+ Add --tres-per-task to salloc/sbatch/srun.
+ Allow nodefeatures plugin features to work with cloud nodes.
e.g. - Powered down nodes have no active changeable features.
- Nodes can't be changed to other active features until powered down.
- Active changeable features are reset/cleared on power down.
+ Make slurmstepd cgroups constrained by total configured memory from
slurm.conf (NodeName=<> RealMemory=#) instead of total physical memory.
+ node_features/helpers - add support for the OR and parentheses operators
in a --constraint expression.
+ slurmctld will fatal() when [Prolog|Epilog]Slurmctld are defined but
are not executable.
+ Validate node registered active features are a super set of node's
currently active changeable features.
+ On clusters without any PrologFlags options, batch jobs with failed
prologs no longer generate an output file.
+ Add SLURM_JOB_START_TIME and SLURM_JOB_END_TIME environment variables.
+ Add SuspendExcStates option to slurm.conf to avoid suspending/powering
down specific node states.
+ Add support for DCMI power readings in IPMI plugin.
+ slurmrestd served /slurm/v0.0.39 and /slurmdb/v0.0.39 endpoints had major
changes from prior versions. Almost all schemas have been renamed and
modified. Sites using OpenAPI Generator clients are highly suggested to
upgrade to to using atleast version 6.x due to limitations with prior
versions.
+ Allow for --nodelist to contain more nodes than required by --nodes.
+ Rename "nodes" to "nodes_resume" in SLURM_RESUME_FILE job output.
+ Rename "all_nodes" to "all_nodes_resume" in SLURM_RESUME_FILE output.
+ Add jobcomp/kafka plugin.
+ Add new PreemptParameters=reclaim_licenses option which will allow higher
priority jobs to preempt jobs to free up used licenses. (This is only
enabled for with PreemptModes of CANCEL and REQUEUE, as Slurm cannot
guarantee suspended jobs will release licenses correctly.)
+ hpe/slingshot - add support for the instant-on feature.
+ Add ability to update SuspendExc* parameters with scontrol.
+ Add ability to restore SuspendExc* parameters on restart with slurmctld
- R option.
+ Add ability to clear a GRES specification by setting it to "0" via
'scontrol update job'.
+ Add SLURM_JOB_OVERSUBSCRIBE environment variable for Epilog, Prolog,
EpilogSlurmctld, PrologSlurmctld, and mail ouput.
+ System node down reasons are appended to existing reasons, separated
by ':'.
+ New command scrun has been added. scrun acts as an Open Container
Initiative (OCI) runtime proxy to run containers seamlessly via Slurm.
+ Fixed GpuFreqDef option. When set in slurm.conf, it will be used if
- -gpu-freq was not explicitly set by the job step.
* Configuration Changes
+ job_container.conf - Added "Dirs" option to list desired private mount
points.
+ node_features plugins - invalid users specified for AllowUserBoot will
now result in fatal() rather than just an error.
+ Deprecate AllowedKmemSpace, ConstrainKmemSpace, MaxKmemPercent, and
MinKmemSpace.
+ Allow jobs to queue even if the user is not in AllowGroups when
EnforcePartLimits=no is set. This ensures consistency for all the
Partition access controls, and matches the documented behavior for
EnforcePartLimits.
+ Add InfluxDBTimeout parameter to acct_gather.conf.
+ job_container/tmpfs - add support for expanding %h and %n in BasePath.
+ slurm.conf - Removed SlurmctldPlugstack option.
+ Add new SlurmctldParameters=validate_nodeaddr_threads=<number> option to
allow concurrent hostname resolution at slurmctld startup.
+ Add new AccountingStoreFlags=job_extra option to store a job's extra field
in the database.
+ Add new "defer_batch" option to SchedulerParameters to only defer
scheduling for batch jobs.
+ Add new DebugFlags option 'JobComp' to replace 'Elasticsearch'.
+ Add configurable job requeue limit parameter - MaxBatchRequeue - in
slurm.conf to permit changes from the old hard-coded value of 5.
+ helpers.conf - Allow specification of node specific features.
+ helpers.conf - Allow many features to one helper script.
+ job_container/tmpfs - Add "Shared" option to support shared namespaces.
This allows autofs to work with the job_container/tmpfs plugin when
enabled.
+ acct_gather.conf - Added EnergyIPMIPowerSensors=Node=DCMI and
Node=DCMI_ENHANCED.
+ Add new "getnameinfo_cache_timeout=<number>" option to
CommunicationParameters to adjust or disable caching the results of
getnameinfo().
+ Add new PrologFlags=ForceRequeueOnFail option to automatically requeue
batch jobs on Prolog failures regardless of the job --requeue setting.
+ Add HealthCheckNodeState=NONDRAINED_IDLE option.
+ Add 'explicit' to Flags in gres.conf. This makes it so the gres is not
automatically added to a job's allocation when --exclusive is used. Note
that this is a per-node flag.
+ Moved the "preempt_" options from SchedulerParameters to
PreemptParameters, and dropped the prefix from the option names.
(The old options will still be parsed for backwards compatibility,
but are now undocumented.)
+ Add LaunchParameters=ulimit_pam_adopt, which enables setting RLIMIT_RSS
in adopted processes.
+ Update SwitchParameters=job_vni to enable/disable creating job VNIs
for all jobs, or when a user requests them.
+ Update SwitchParameters=single_node_vni to enable/disable creating
single node vnis for all jobs, or when a user requests them.
+ Add ability to preserve SuspendExc* parameters on reconfig with
ReconfigFlags=KeepPowerSaveSettings.
+ slurmdbd.conf - Add new AllResourcesAbsolute to force all new resources
to be created with the Absolute flag.
+ topology/tree - Add new TopologyParam=SwitchAsNodeRank option to reorder
nodes based on switch layout. This can be useful if the naming convention
for the nodes does not natually map to the network topology.
+ Removed the default setting for GpuFreqDef. If unset, no attempt to change
the GPU frequency will be made if --gpu-freq is not set for the step.
* Command Changes
+ sacctmgr - no longer force updates to the AdminComment, Comment, or
SystemComment to lower-case.
+ sinfo - Add -F/--future option to sinfo to display future nodes.
+ sacct - Rename 'Reserved' field to 'Planned' to match sreport and the
nomenclature of the 'Planned' node.
+ scontrol - advanced reservation flag MAINT will no longer replace nodes,
similar to STATIC_ALLOC
+ sbatch - add parsing for #PBS -d and #PBS -w.
+ scontrol show assoc_mgr will show username(uid) instead of uid in
QoS section.
+ Add strigger --draining and -R/--resume options.
+ Change --oversubscribe and --exclusive to be mutually exclusive for
job submission. Job submission commands will now fatal if both are set.
Previously, these options would override each other, with the last one
in the job submission command taking effect.
+ scontrol - Requested TRES and allocated TRES will now always be printed
when showing jobs, instead of one TRES output that was either the
requested or allocated.
+ srun --ntasks-per-core now applies to job and step allocations. Now,
use of --ntasks-per-core=1 implies --cpu-bind=cores and
- -ntasks-per-core>1 implies --cpu-bind=threads.
+ salloc/sbatch/srun - Check and abort if ntasks-per-core >
threads-per-core.
+ scontrol - Add ResumeAfter=<secs> option to "scontrol update nodename=".
+ Add a new "nodes=" argument to scontrol setdebug to allow the debug
level on the slurmd processes to be temporarily altered.
+ Add a new "nodes=" argument to "scontrol setdebugflags" as well.
+ Make it so scrontab prints client-side the job_submit() err_msg (which
can be set i.e. by using the log_user() function for the lua plugin).
+ scontrol - Reservations will not be allowed to have STATIC_ALLOC or
MAINT flags and REPLACE[_DOWN] flags simultaneously.
+ scontrol - Reservations will only accept one reoccurring flag when
being created or updated.
+ scontrol - A reservation cannot be updated to be reoccurring if it is
already a floating reservation.
+ squeue - removed unused '%s' and 'SelectJobInfo' formats.
+ squeue - align print format for exit and derived codes with that of
other components (<exit_status>:<signal_number>).
+ sacct - Add --array option to expand job arrays and display array
tasks on separate lines.
+ Partial support for '--json' and '--yaml' formated outputs have been
implemented for sacctmgr, sdiag, sinfo, squeue, and scontrol. The
resultant data ouput will be filtered by normal command arguments.
Formatting arguments will continue to be ignored.
+ salloc/sbatch/srun - extended the --nodes syntax to allow for a list
of valid node counts to be allocated to the job. This also supports
a "step count" value (e.g., --nodes=20-100:20 is equivalent to
- -nodes=20,40,60,80,100) which can simplify the syntax when the job
needs to scale by a certain "chunk" size.
+ srun - add user requestible vnis with '--network=job_vni' option.
+ srun - add user requestible single node vnis with the
'--network=single_node_vni' option.
* API Changes
+ job_container plugins - container_p_stepd_create() function signature
replaced uint32_t uid with stepd_step_rec_t* step.
+ gres plugins - gres_g_get_devices() function signature replaced pid_t
pid with stepd_step_rec_t* step.
+ cgroup plugins - task_cgroup_devices_constrain() function signature
removed pid_t pid.
+ task plugins - replace task_p_pre_set_affinity(), task_p_set_affinity(),
and task_p_post_set_affinity() with task_p_pre_launch_priv() like it
was back in slurm 20.11.
+ Allow for concurrent processing of job_submit_g_submit() and
job_submit_g_modify() calls. If your plugin is not capable of concurrent
operation you must add additional locking within your plugin.
+ Removed return value from slurm_list_append().
+ The List and ListIterator types have been removed in favor of list_t
and list_itr_t respectively.
+ burst buffer plugins - add bb_g_build_het_job_script().
bb_g_get_status() - added authenticated UID and GID.
bb_g_run_script() - added job_info argument.
+ burst_buffer.lua - Pass UID and GID to most hooks. Pass job_info
(detailed job information) to many hooks. See
etc/burst_buffer.lua.example for a complete list of changes.
WARNING: Backwards compatibility is broken for
slurm_bb_get_status: UID and GID are passed before the variadic
arguments. If UID and GID are not explicitly listed as arguments to
slurm_bb_get_status(), then they will be included in the variadic
arguments.
Backwards compatibility is maintained for all other hooks because
the new arguments are passed after the existing arguments.
+ node_features plugins - node_features_p_reboot_weight() function
removed.
node_features_p_job_valid() - added parameter feature_list.
node_features_p_job_xlate() - added parameters feature_list and
job_node_bitmap.
+ New data_parser interface with v0.0.39 plugin.
* Added: Fix-test-1.99.patch
* Reworked: Fix-test-38.11.patch
pam_slurm-Initialize-arrays-and-pass-sizes.patch
* Thu Feb 09 2023 Egbert Eich <eich@suse.com>
- testsuite: on later SUSE versions claim ownership of directory
/etc/security/limits.d.

Files

/usr/lib64/slurm
/usr/lib64/slurm/accounting_storage_mysql.so
/usr/lib64/slurm/jobcomp_mysql.so

Generated by rpm2html 1.8.1

Fabrice Bellet, Sun Apr 19 22:35:28 2026

slurm-sql-25.11.2-1.2 RPM for x86_64

From OpenSuSE Tumbleweed for x86_64

Provides

Requires

License

Changelog

Files